Data query method, device, apparatus and medium

Abstract

The application provides a data query method and device, equipment and medium. The method comprises the following steps: based on a plurality of business data to be queried, a multi-path search balanced lookup tree is constructed according to at least one relational attribute, efficient attribute range query can be realized, the adaptability and query efficiency in different business scenarios are improved, the multi-path search balanced lookup tree can realize dynamic data updating, and the maintenance effect of data query is improved; for each node of the multi-path search balanced lookup tree, a vector index mode used by the node is determined based on the selection degree of the node, the configuration of vector index resources is effectively improved by dynamically constructing the vector index mode, memory waste or performance bottleneck problems caused by uniform vector index are avoided, and the accuracy of data query is improved; through the combined query of the multi-path search balanced lookup tree and the vector index, the performance of data query is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a data query method, apparatus, device, and medium. Background Technology

[0002] Against the backdrop of the rapid development of big data and artificial intelligence technologies, range-filtering approximate nearest neighbor (RFANN) queries have become an important research direction in the field of information retrieval. RFANN queries can efficiently find the most similar data object to a given query vector among data objects that satisfy specific attribute range constraints, and are widely used in recommendation systems, image retrieval, semantic search, product matching, and other scenarios.

[0003] How to implement efficient index structures and query strategies to support hybrid queries that combine range filtering and near nearest neighbor search has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, this application provides a data query method, apparatus, device and medium that improves the performance of data query by combining multi-way search balanced search tree and vector index for joint query.

[0005] Specifically, this application is implemented through the following technical solution:

[0006] According to a first aspect of this application, a data query method is provided, the method comprising:

[0007] Based on multiple business data to be queried, a multi-way search balanced search tree is constructed according to at least one relational attribute; in the multi-way search balanced search tree, each non-leaf node is used to indicate the corresponding attribute range, and each leaf node is used to store the business data belonging to the attribute range indicated by the parent node of that leaf node.

[0008] For each node of the multi-way search balanced search tree, a vector indexing method is determined for the node based on the node's selectivity; the selectivity indicates the frequency with which the node is accessed during the query process; the vector indexing method indicates whether there is an index or not, and the type of vector index when there is an index.

[0009] In response to a query request, target business data is selected from the multiple business data based on the query attribute range and query vector corresponding to the query request, the attribute range corresponding to each non-leaf node of the multi-way search balanced search tree, and the vector indexing method corresponding to each node.

[0010] In one optional implementation, the step of filtering target business data from the multiple business data based on the query attribute range and query vector corresponding to the query request, the attribute range corresponding to each non-leaf node of the multi-way search balanced search tree, and the vector indexing method corresponding to each node includes:

[0011] Based on the query attribute range and the attribute range corresponding to each non-leaf node of the multi-way search balanced search tree, at least one target node is selected from the multi-way search balanced search tree.

[0012] Based on the query vector, the business data corresponding to each target node is queried according to the vector indexing method corresponding to each target node to determine the target business data.

[0013] In one optional implementation, the step of querying the business data corresponding to each target node based on the query vector and according to the vector indexing method corresponding to each target node to determine the target business data includes:

[0014] For each target node, based on the query vector, the business data corresponding to the target node is queried according to the vector indexing method corresponding to the target node, and a first preset number of candidate business data with the highest similarity to the query vector are determined;

[0015] The candidate service data with the highest similarity among the candidate service data corresponding to each target node are determined as the target service data.

[0016] In one optional implementation, the step of querying the business data corresponding to the target node based on the query vector and according to the vector indexing method corresponding to the target node, and determining a first preset number of candidate business data with the highest similarity to the query vector, includes:

[0017] If the vector indexing method corresponding to the target node is no index, a K nearest neighbor query is performed on the business data corresponding to the target node based on the query vector to determine a first preset number of candidate business data with the highest similarity to the query vector.

[0018] If the vector indexing method corresponding to the target node is indexed, based on the query vector and according to the vector indexing type indicated by the vector indexing method, an approximate nearest neighbor query is performed on the business data corresponding to the target node to determine a first preset number of candidate business data with the highest similarity to the query vector.

[0019] In one optional implementation, determining the vector indexing method for the node based on its selectivity includes:

[0020] If the selectivity of a node is greater than a first threshold, the node is determined to use an indexed vector indexing method, and the corresponding vector index type is a graph index.

[0021] If the selectivity of a node is greater than or equal to the second threshold and less than or equal to the first threshold, it is determined that the vector indexing method used by the node is indexed and the corresponding vector index type is clustering index.

[0022] If the selectivity of a node is less than the second threshold, the vector indexing method used by the node is determined to be no index.

[0023] In one optional implementation, the selectivity of the node is determined by the following steps:

[0024] The selectivity of a node is determined based on the ratio of the amount of business data corresponding to that node to the total amount of business data.

[0025] In one optional implementation, the method further includes:

[0026] Upon receiving a vector index reconstruction request, the selectivity of each node in the multi-way search balanced search tree is redefined;

[0027] Adjust the vector indexing method used by the node based on the updated selectivity.

[0028] According to a second aspect of this application, a data query apparatus is provided, the apparatus comprising:

[0029] The tree construction module is used to construct a multi-way search balanced search tree based on multiple business data to be queried, according to at least one relational attribute; in the multi-way search balanced search tree, each non-leaf node is used to indicate the corresponding attribute range, and each leaf node is used to store the business data belonging to the attribute range indicated by the parent node of that leaf node.

[0030] The index determination module is used to determine the vector indexing method for each node of the multi-way search balanced search tree based on the selectivity of the node; the selectivity is used to indicate the frequency of the node being accessed during the query process; the vector indexing method is used to indicate whether there is an index or not, and the type of vector index when there is an index.

[0031] The data filtering module is used to respond to a query request and filter out target business data from the multiple business data based on the query attribute range and query vector corresponding to the query request, the attribute range corresponding to each non-leaf node of the multi-way search balanced search tree, and the vector indexing method corresponding to each node.

[0032] According to a third aspect of this application, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the data query method described in the first aspect above.

[0033] According to a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the data query method described in the first aspect above.

[0034] The data query method, apparatus, device, and medium provided in this application embodiment construct a multi-way search balanced search tree based on multiple business data to be queried, according to at least one relational attribute. This enables efficient attribute range queries, improves adaptability and query efficiency in different business scenarios, and allows for dynamic data updates, which helps improve the maintenance effect of data queries. For each node of the multi-way search balanced search tree, the vector index method used for the node is determined based on the node's selectivity. By dynamically constructing the vector index method, the configuration of vector index resources is effectively improved, avoiding memory waste or performance bottlenecks caused by a unified vector index, which helps improve the accuracy of data queries. The joint query combining the multi-way search balanced search tree and the vector index helps improve the performance of data queries.

[0035] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure.

[0036] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating a data query method in an exemplary embodiment of this application;

[0038] Figure 2 This is a schematic diagram illustrating a multi-way search balanced search tree according to an exemplary embodiment of this application;

[0039] Figure 3 This is a schematic diagram illustrating a data query process according to an exemplary embodiment of this application;

[0040] Figure 4 This is a schematic diagram of a data query device shown in an exemplary embodiment of this application;

[0041] Figure 5This is a schematic diagram of the structure of a computer device shown in an exemplary embodiment of this application. Detailed Implementation

[0042] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0043] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0044] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0045] Research has revealed that RFANN queries combine attribute range constraints with high-dimensional vector similarity matching for approximate nearest neighbor queries. The goal of an RFANN query is to efficiently find the vector object most similar to a given query vector among data objects that satisfy specific attribute range constraints. This not only finds the object most similar to the query vector but also ensures that the found object meets the range constraints of one or more numerical attributes. For example, in e-commerce platforms, users might want to find products similar to uploaded images while limiting the price range; in video retrieval systems, users might want to find video clips semantically similar to the query content while limiting the timestamp range. These applications typically require the system to quickly locate approximate nearest neighbors in a high-dimensional vector space while satisfying the range constraints of one or more numerical attributes, posing a dual challenge to query efficiency and accuracy. Therefore, how to implement efficient index structures and query strategies to support hybrid queries that balance range filtering and vector approximate retrieval has become a pressing issue in current database and information retrieval system research.

[0046] When performing RFANN queries, pre-filtering or post-filtering is often used.

[0047] Pre-filtering involves first filtering the dataset based on attribute range constraints, selecting data objects that meet the query range to form a smaller candidate dataset. Then, a search is performed on this candidate dataset, such as a K-Nearest Neighbors (KNN) search, to obtain the final results. This method typically uses binary search or similar techniques to quickly filter out data objects that meet the query range, making it suitable for low-dimensional attribute range filtering scenarios.

[0048] Post-filtering involves first performing a search on the entire dataset, such as an Approximate Nearest Neighbor (ANN) search, to obtain a set of candidate data objects. Then, a query range constraint is applied to these candidate data objects, retaining only the results that satisfy the query range as the final result. Compared to pre-filtering, this approach avoids problems such as information loss caused by prematurely narrowing the search space and is suitable for attribute dimensions with a small scope that are difficult to index efficiently.

[0049] However, with pre-filtering, if the query range is large, the amount of candidate dataset obtained is large, which leads to low efficiency in subsequent retrieval. In addition, since pre-filtering selects data objects that meet the query range conditions first, objects outside the query range are excluded. In some cases, this may result in the inability to find the required data, thus affecting the recall rate. Especially in high-dimensional space, data sparsity makes many data objects lack effective adjacency points within the query range, affecting the quality of data query.

[0050] In the post-filtering approach, after applying query range constraints to candidate data objects, the number of final results may be unstable, or even none of the candidate data objects may meet the query range, resulting in empty final results and thus affecting the recall rate.

[0051] Furthermore, pre-filtering and post-filtering lack the ability to dynamically adjust the index structure based on data distribution when facing different query ranges, which can easily lead to wasted resources or decreased query efficiency. In addition, most current index structures are static, making it difficult to adapt to dynamic data updates.

[0052] Based on the above research, this application provides a data query method, apparatus, device, and medium. The combined query using a multi-way search balanced search tree and vector index helps improve data query performance. Furthermore, the multi-way search balanced search tree enables dynamic data updates, improving data query maintenance. For each node in the multi-way search balanced search tree, the vector index method used is determined based on the node's selectivity. By dynamically constructing the vector index method, the configuration of vector index resources is effectively improved, avoiding memory waste or performance bottlenecks caused by a unified vector index, thus contributing to improved data query accuracy.

[0053] To facilitate understanding of this embodiment, a data query method disclosed in this application will first be described in detail. The execution entity of the data query method provided in this application is generally an electronic device with a certain computing power. This electronic device can be a server, which can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data, and artificial intelligence platforms. In some possible implementations, this data query method can be implemented by a processor calling computer-readable instructions stored in memory.

[0054] The following description, in conjunction with the accompanying drawings, illustrates a data query method provided by an embodiment of this application.

[0055] See Figure 1 The diagram shown is a flowchart illustrating a data query method according to an exemplary embodiment of this application. Figure 1 As shown in the figure, the data query method provided in this embodiment includes steps S101 to S103, wherein:

[0056] S101: Based on multiple business data to be queried, construct a multi-way search balanced search tree according to at least one relational attribute; in the multi-way search balanced search tree, each non-leaf node is used to indicate the corresponding attribute range, and each leaf node is used to store the business data belonging to the attribute range indicated by the parent node of that leaf node.

[0057] Here, the multiple business data to be queried can be data from corresponding business scenarios. For example, in an e-commerce shopping scenario, the multiple business data to be queried can be multiple product data; in a video retrieval scenario, the multiple business data to be queried can be multiple video data. Optionally, the multiple business data to be queried can be stored in a database.

[0058] Among them, the relational attributes are scalars, that is, numerical attributes without direction, such as price, time, location, rating, etc. Taking time as an example, it can be further divided into online time, listing time, etc.

[0059] It is understood that the multiple business data to be queried in the database may be unordered. The process of constructing a multi-way search balanced search tree is the process of recursively dividing the data interval according to at least one relational attribute, thereby forming a balanced multi-level tree structure. In this way, by constructing a multi-way search balanced search tree, the organization and division of attribute intervals can be automatically realized without the need for data sorting preprocessing, which helps to improve the efficiency of data query.

[0060] Here, the multi-way search balanced search tree is specifically a B-tree. In this tree, each non-leaf node indicates a corresponding attribute range, and each leaf node stores the business data for the attribute range indicated by its parent node. This hierarchical division of the multi-way search balanced search tree ensures that queries within any range can be quickly located among several matching nodes, thus providing a structural foundation for subsequent vector retrieval. It can be understood that leaf nodes are nodes without child nodes, and non-leaf nodes are nodes with child nodes.

[0061] Furthermore, the multi-way search balanced search tree has good dynamic adaptability, enabling dynamic data updates and supporting subsequent data update operations such as insertion and deletion, which helps improve the maintenance effect of data query.

[0062] Here, the number of business data corresponding to each node at the same level in the multi-way search balanced search tree is basically the same, so the query speed of different branches is similar, which helps to improve the performance and stability of data query.

[0063] For example, see [link / reference] Figure 2 This is a schematic diagram illustrating a multi-way search balanced search tree, as shown in an exemplary embodiment of this application. Figure 2As shown in the diagram, taking the price as an example of a relational attribute, the price range corresponding to the multiple business data to be queried is 1-40 yuan. The root node of the constructed multi-way search balanced search tree is used to indicate the attribute range of 1-40. The second level of the multi-way search balanced search tree includes two nodes, used to indicate the attribute ranges of 1-20 and 20-40 respectively. The third level of the multi-way search balanced search tree includes four nodes, used to indicate the attribute ranges of 1-10, 10-20, 20-30, and 30-40 respectively. In this structure, the two nodes indicating the attribute ranges 1-10 and 10-20 are child nodes indicating the attribute range 1-20, and the two nodes indicating the attribute ranges 20-30 and 30-40 are child nodes indicating the attribute range 20-40. The fourth level of the multi-way search balanced search tree includes eight nodes, respectively indicating the attribute ranges 1-5, 5-10, 10-15, 15-20, 20-25, 25-30, 30- The attribute range is defined as 35 and the attribute range is 35-40. Specifically, the two nodes indicating the attribute ranges 1-5 and 5-10 are child nodes indicating the attribute range 1-10; the two nodes indicating the attribute ranges 10-15 and 15-20 are child nodes indicating the attribute range 10-20; the two nodes indicating the attribute ranges 20-25 and 25-30 are child nodes indicating the attribute range 20-30; and the nodes indicating the attribute ranges 30-35 and 35-40... The two nodes are child nodes used to indicate the attribute range of 30-40; the fifth level of the multi-way search balanced search tree includes eight nodes, which are used to store business data belonging to the attribute range of 1-5, the attribute range of 5-10, the attribute range of 10-15, the attribute range of 15-20, the attribute range of 20-25, the attribute range of 25-30, the attribute range of 30-35, and the attribute range of 35-40, respectively.

[0064] S102: For each node of the multi-way search balanced search tree, based on the selectivity of the node, determine the vector indexing method to be used for the node; the selectivity is used to indicate the frequency of the node being accessed during the query process; the vector indexing method is used to indicate whether there is an index or not, and the type of vector index when there is an index.

[0065] In this step, for each node of the multi-way search balanced search tree, the selectivity of the node can be determined. Based on the selectivity of the node, the vector indexing method to be used for the node is determined, which helps to improve the configuration effect of index resources.

[0066] In some possible implementations, the selectivity of the node is determined by the following steps:

[0067] The selectivity of a node is determined based on the ratio of the amount of business data corresponding to that node to the total amount of business data.

[0068] In this step, the selectivity of the node can be determined as the ratio of the number of business data corresponding to the node to the total number of business data.

[0069] When determining the amount of business data corresponding to a node, optionally, the amount of business data corresponding to the node can be accurately counted based on the multi-way search balanced search tree. This can improve the accuracy of the selection degree determination, thereby improving the accuracy of the vector indexing method.

[0070] As described above, the multiple business data to be queried can be stored in a database. The data in the database can be pre-divided according to preset attribute intervals, and the number of business data belonging to each preset attribute interval can be counted. Optionally, when determining the number of business data corresponding to a node, the number of business data corresponding to the node can be determined based on the attribute interval corresponding to the node and the number of business data in each preset attribute interval. Thus, by using the pre-determined number of business data belonging to each preset attribute interval to determine the number of business data corresponding to the node, the efficiency of data querying can be improved.

[0071] In some possible implementations, determining the vector indexing method for the node based on its selectivity includes:

[0072] If the selectivity of a node is greater than a first threshold, the node is determined to use an indexed vector indexing method, and the corresponding vector index type is a graph index.

[0073] If the selectivity of a node is greater than or equal to the second threshold and less than or equal to the first threshold, it is determined that the vector indexing method used by the node is indexed and the corresponding vector index type is clustering index.

[0074] If the selectivity of a node is less than the second threshold, the vector indexing method used by the node is determined to be no index.

[0075] Here, if the selectivity of the node is greater than the first threshold, it means that the attribute range corresponding to the node is large and the business data corresponding to the node is more likely to be queried during the query process. Therefore, it is suitable to build a high-quality vector index type. In this embodiment of the disclosure, the vector index method used by the node is determined to be indexed, and the corresponding vector index type is a graph index, such as a hierarchical navigable small world (HNSW). This can utilize the efficient neighborhood jumping capability of the graph index to improve the accuracy of data query.

[0076] If the selectivity of a node is greater than or equal to the second threshold and less than or equal to the first threshold, it indicates that the attribute range corresponding to the node is moderate, and therefore it is suitable to build a lightweight index to save resources. In this embodiment of the disclosure, the vector indexing method used by the node is determined to be indexed, and the corresponding vector index type is a clustering index, such as an inverted file (IVF). This can combine the inverted index mechanism and the cluster center to optimize search efficiency, while taking into account memory usage and query performance.

[0077] If the selectivity of a node is less than the second threshold, it means that the attribute range corresponding to the node is small and the business data corresponding to the node is less likely to be queried during the query process. Therefore, it is suitable to use an indexless approach, which can save resources and avoid unnecessary index overhead.

[0078] The specific values ​​of the first threshold and the second threshold can be set according to actual needs, and are not specifically limited here. For example, the first threshold can be set to 10% and the second threshold to 1%.

[0079] In this way, the vector indexing method used for nodes is dynamically determined according to different selectivity, which effectively improves the configuration of vector index resources, avoids memory waste or performance bottlenecks caused by uniform vector indexes, prevents performance degradation caused by using inefficient vector indexes in high selectivity scenarios and memory waste caused by using efficient vector indexes in low selectivity scenarios, and helps to improve the efficiency of data querying under different selectivity.

[0080] For example, see again Figure 2 In this example, the root node of the multi-way search balanced search tree uses an indexed vector index, and the corresponding vector index type is graph index; the nodes in the second and third levels of the multi-way search balanced search tree use an indexed vector index, and the corresponding vector index type is cluster index; the nodes in the fourth level of the multi-way search balanced search tree use an unindexed vector index.

[0081] S103: In response to a query request, based on the query attribute range and query vector corresponding to the query request, the attribute range corresponding to each non-leaf node of the multi-way search balanced search tree, and the vector indexing method corresponding to each node, the target business data is filtered out from the multiple business data.

[0082] In this step, in response to a query request, the query attribute range and query vector carried in the query request are determined. Based on the query attribute range, the query vector, the attribute range corresponding to each non-leaf node of the multi-way search balanced search tree, and the vector indexing method corresponding to each node, the target business data is filtered out from the multiple business data.

[0083] For example, in an e-commerce shopping scenario, if a user wants to find products similar to uploaded images and specifies a price range, the query attribute range can be the user-defined price range, and the query vector can be the product image uploaded by the user. In a video retrieval scenario, if a user wants to find videos semantically similar to the query content and specifies a timestamp range, the query attribute range can be the user-defined timestamp range, and the query vector can be the query content uploaded by the user.

[0084] In some possible implementations, the step of filtering target business data from the multiple business data based on the query attribute range and query vector corresponding to the query request, the attribute range corresponding to each non-leaf node of the multi-way search balanced search tree, and the vector indexing method corresponding to each node includes:

[0085] Based on the query attribute range corresponding to the query request and the attribute range corresponding to each non-leaf node of the multi-way search balanced search tree, at least one target node is selected from the multi-way search balanced search tree.

[0086] Based on the query vector corresponding to the query request, the business data corresponding to each target node is queried according to the vector indexing method corresponding to each target node to determine the target business data.

[0087] In the above steps, following the hierarchical structure of the multi-way search balanced search tree, starting from the root node, all nodes that intersect with the query attribute interval can be recursively searched, and the union of the attribute intervals corresponding to all target nodes obtained by filtering is the query attribute interval.

[0088] For example, see again Figure 2For example, if the query attribute range is 1-30, the filtered target nodes include nodes indicating the attribute range of 1-10, nodes indicating the attribute range of 10-20, and nodes indicating the attribute range of 20-30; if the query attribute range is 1-25, the filtered target nodes include nodes indicating the attribute range of 1-10, nodes indicating the attribute range of 10-20, and nodes indicating the attribute range of 20-25.

[0089] After the target nodes are obtained through filtering, node-level vectors can be used. Based on the query vector corresponding to the query request, the business data corresponding to each target node can be queried according to the vector indexing method corresponding to each target node to determine the target business data.

[0090] In this way, by decomposing the query attribute range, the range search capability of the multi-way search balanced search tree can be fully utilized, thereby ensuring that each target node matching the query attribute range can be quickly located. Based on the query vector corresponding to the query request, the query is performed according to the vector index method corresponding to each target node, which helps to improve the efficiency of data query.

[0091] In some possible implementations, the step of querying the business data corresponding to each target node based on the query vector corresponding to the query request and according to the vector indexing method corresponding to each target node to determine the target business data includes:

[0092] For each target node, based on the query vector corresponding to the query request, the business data corresponding to the target node is queried according to the vector indexing method corresponding to the target node, and a first preset number of candidate business data with the highest similarity to the query vector are determined.

[0093] The candidate service data with the highest similarity among the candidate service data corresponding to each target node are determined as the target service data.

[0094] In the above steps, an independent vector retrieval is performed for each target node. Based on the query vector corresponding to the query request, the business data corresponding to the target node is queried according to the vector indexing method corresponding to the target node, and a first preset number of candidate business data with the highest similarity to the query vector are determined.

[0095] Here, when determining the similarity between business data and the query vector, the distance between the business data and the query vector can be determined, and the distance can be used as the similarity.

[0096] Optionally, the process of querying each target node to determine candidate business data can be performed in parallel using multi-threaded queries, which helps to improve the throughput and response speed of data queries.

[0097] After determining the candidate service data corresponding to each target node, the candidate service data corresponding to each target node can be merged and sorted according to the similarity of the candidate service data, such as sorting from small to large or from large to small, and the second preset number of candidate service data with the highest similarity can be selected as the target service data.

[0098] The specific values ​​of the first preset quantity and the second preset quantity can be determined according to the actual candidate node optimization needs, and no specific limitation is made here. The second preset quantity and the first preset quantity can be the same or different.

[0099] In this way, for complex range queries across nodes, the approach of querying each target node separately and merging and reordering the query results of each node ensures the flexibility of data query while taking into account performance and implementation complexity, which helps to improve recall and query quality.

[0100] In some possible implementations, the step of querying the business data corresponding to the target node based on the query vector corresponding to the query request, according to the vector indexing method corresponding to the target node, and determining a first preset number of candidate business data with the highest similarity to the query vector, includes:

[0101] If the vector indexing method corresponding to the target node is no index, a K-nearest neighbor query is performed on the business data corresponding to the target node based on the query vector corresponding to the query request to determine a first preset number of candidate business data with the highest similarity to the query vector.

[0102] If the vector indexing method corresponding to the target node is indexed, based on the query vector corresponding to the query request, and according to the vector indexing type indicated by the vector indexing method, an approximate nearest neighbor query is performed on the business data corresponding to the target node to determine a first preset number of candidate business data with the highest similarity to the query vector.

[0103] In the above steps, when the vector index method corresponding to the target node is unindexed, vector matching is performed directly using the K-Nearest Neighbors (KNN) query method based on the query vector to determine a first preset number of candidate business data with the highest similarity to the query vector; when the vector index method corresponding to the target node is indexed, an Approximate Nearest Neighbor (ANN) query is performed on the business data corresponding to the target node based on the query vector and according to the vector index type indicated by the vector index method to determine a first preset number of candidate business data with the highest similarity to the query vector.

[0104] For example, following the above example, please refer again. Figure 2 Taking the selected target nodes as an example, which include nodes indicating the attribute range of 1-10, nodes indicating the attribute range of 10-20, and nodes indicating the attribute range of 20-30, the vector indexing method used by these three nodes is indexed, and the corresponding vector index type is clustering index. Therefore, for each node, based on the query vector, an ANN query can be performed on the business data corresponding to the target node according to the clustering index to determine the first preset number of candidate business data with the highest similarity to the query vector. Taking the selected target nodes as an example, which include nodes indicating attribute ranges of 1-10, 10-20, and 20-25, the nodes indicating attribute ranges of 1-10 and 10-20 use an indexed vector index, and the corresponding vector index type is a clustering index. The nodes indicating attribute ranges of 20-25 use an unindexed vector index. Therefore, for the nodes indicating attribute ranges of 1-10 and 10-20, an ANN query can be performed on the business data corresponding to the target nodes based on the query vector and the clustering index to determine a first preset number of candidate business data with the highest similarity to the query vector. For the nodes indicating attribute ranges of 20-25, a KNN query can be directly used to perform vector matching based on the query vector to determine a first preset number of candidate business data with the highest similarity to the query vector.

[0105] In this way, while ensuring the flexibility and efficiency of data query, it is applicable to data queries of nodes with different selectivity, which has broad engineering application value and helps to improve the recall rate and query efficiency under range filtering conditions.

[0106] For a clearer illustration of the data query process, see [link to relevant documentation]. Figure 3This is a schematic diagram illustrating a data query process as shown in an exemplary embodiment of this application. Figure 3 As shown, based on multiple business data to be queried, a multi-way search balanced search tree is constructed according to at least one relational attribute; for each node of the multi-way search balanced search tree, a vector indexing method is determined for the node based on the node's selectivity; in response to a query request, the target business data is filtered from the multiple business data based on the query attribute range and query vector corresponding to the query request, the attribute ranges corresponding to each non-leaf node of the multi-way search balanced search tree, and the vector indexing method corresponding to each node. Specific steps are described in the foregoing embodiments and will not be repeated here.

[0107] In some possible implementations, the method further includes:

[0108] Upon receiving a vector index reconstruction request, the selectivity of each node in the multi-way search balanced search tree is redefined;

[0109] Adjust the vector indexing method used by the node based on the updated selectivity.

[0110] Here, the multi-way search balanced search tree has good dynamic adaptability and can realize dynamic data updates. Correspondingly, receiving a vector index reconstruction request includes detecting an update to the multi-way search balanced search tree, or reaching the time for the periodic vector index reconstruction of the multi-way search balanced search tree.

[0111] The multi-way search balanced search tree update includes inserting new business data into the multi-way search balanced search tree, deleting business data from the multi-way search balanced search tree, and changing the relational attribute corresponding to any business data in the multi-way search balanced search tree.

[0112] It is understood that inserting new business data into the multi-way search balanced search tree can lead to node splitting, deleting business data from the multi-way search balanced search tree can lead to node merging, and changing the relational attribute corresponding to any business data in the multi-way search balanced search tree can lead to node deletion and insertion. All of these will cause changes to the structure of the multi-way search balanced search tree. In order to ensure the accuracy of data query, the selectivity of each node in the multi-way search balanced search tree can be re-determined.

[0113] For example, in an e-commerce shopping scenario, if a product was originally priced at 40 yuan and is now reduced to 10 yuan, the business data needs to be deleted from the original node and inserted into the new node.

[0114] Specifically, adjusting the vector indexing method used by the node based on the updated selectivity includes: determining whether the vector indexing method currently used by the node needs to be changed based on the comparison result between the first threshold and the updated selectivity, and the comparison result between the second threshold and the updated selectivity. If a change is needed, the updated vector indexing method can be determined; if no change is needed, the original vector indexing method can be maintained.

[0115] For example, taking a first threshold of 10% and a second threshold of 1%, if the selectivity of a node decreases from 12% to 0.5%, the previously used indexed vector index (which was a graph index) can be changed to an indexless index to avoid resource waste caused by high-cost indexes. Conversely, when the selectivity of a node increases from 0.5% to 12%, the previously used indexless index can be changed to an indexed vector index (which was a graph index) to improve query performance. This dynamic adjustment mechanism effectively avoids the performance degradation caused by the mismatch between the index structure and the data distribution.

[0116] In this way, when a vector index reconstruction request is received, the vector index method used by the node can be automatically reconstructed, realizing dynamic adaptation of the vector index method. This achieves optimal resource allocation while maintaining high query performance, which helps to enhance the adaptability of data query to changes in business data and ensures the stability of data query in dynamic data environments.

[0117] The data query method provided in this application constructs a multi-way search balanced search tree based on multiple business data to be queried, according to at least one relational attribute. This enables efficient attribute range queries, improves adaptability and query efficiency in different business scenarios, and allows for dynamic data updates, which helps improve the maintenance effect of data queries. For each node of the multi-way search balanced search tree, the vector index method used for the node is determined based on the node's selectivity. By dynamically constructing the vector index method, the configuration of vector index resources is effectively improved, avoiding memory waste or performance bottlenecks caused by a unified vector index. The joint query combining the multi-way search balanced search tree and the vector index helps improve the data query effect.

[0118] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0119] Corresponding to the aforementioned embodiments of the data query method, this application also provides embodiments of the data query device.

[0120] Please see Figure 4 This is a schematic diagram illustrating a data query device according to an exemplary embodiment of this application. Figure 4 As shown in the figure, the data query device 400 provided in this application embodiment includes:

[0121] Tree construction module 401 is used to construct a multi-way search balanced search tree based on multiple business data to be queried, according to at least one relational attribute; in the multi-way search balanced search tree, each non-leaf node is used to indicate the corresponding attribute range, and each leaf node is used to store the business data belonging to the attribute range indicated by the parent node of that leaf node.

[0122] The index determination module 402 is used to determine the vector indexing method to be used for each node of the multi-way search balanced search tree based on the selectivity of the node; the selectivity is used to indicate the frequency of the node being accessed during the query process; the vector indexing method is used to indicate whether there is an index or not, and the type of vector index when there is an index.

[0123] The data filtering module 403 is used to respond to a query request and filter out target business data from the multiple business data based on the query attribute range and query vector corresponding to the query request, the attribute range corresponding to each non-leaf node of the multi-way search balanced search tree, and the vector indexing method corresponding to each node.

[0124] In some possible implementations, when the data filtering module 403 filters target business data from the plurality of business data based on the query attribute range and query vector corresponding to the query request, the attribute range corresponding to each non-leaf node of the multi-way search balanced search tree, and the vector index method corresponding to each node, it is specifically used for:

[0125] Based on the query attribute range corresponding to the query request and the attribute range corresponding to each non-leaf node of the multi-way search balanced search tree, at least one target node is selected from the multi-way search balanced search tree.

[0126] Based on the query vector corresponding to the query request, the business data corresponding to each target node is queried according to the vector indexing method corresponding to each target node to determine the target business data.

[0127] In some possible implementations, when the data filtering module 403 queries the business data corresponding to each target node based on the query vector corresponding to the query request and according to the vector indexing method corresponding to each target node to determine the target business data, it is specifically used for:

[0128] For each target node, based on the query vector corresponding to the query request, the business data corresponding to the target node is queried according to the vector indexing method corresponding to the target node, and a first preset number of candidate business data with the highest similarity to the query vector are determined.

[0129] The candidate service data with the highest similarity among the candidate service data corresponding to each target node are determined as the target service data.

[0130] In some possible implementations, when the data filtering module 403 queries the business data corresponding to the target node based on the query vector corresponding to the query request and according to the vector indexing method corresponding to the target node, and determines a first preset number of candidate business data with the highest similarity to the query vector, it is specifically used for:

[0131] If the vector indexing method corresponding to the target node is no index, a K-nearest neighbor query is performed on the business data corresponding to the target node based on the query vector corresponding to the query request to determine a first preset number of candidate business data with the highest similarity to the query vector.

[0132] If the vector indexing method corresponding to the target node is indexed, based on the query vector corresponding to the query request, and according to the vector indexing type indicated by the vector indexing method, an approximate nearest neighbor query is performed on the business data corresponding to the target node to determine a first preset number of candidate business data with the highest similarity to the query vector.

[0133] In some possible implementations, the index determination module 402, when determining the vector indexing method for the node based on the node's selectivity, specifically performs the following:

[0134] If the selectivity of a node is greater than a first threshold, the node is determined to use an indexed vector indexing method, and the corresponding vector index type is a graph index.

[0135] If the selectivity of a node is greater than or equal to the second threshold and less than or equal to the first threshold, it is determined that the vector indexing method used by the node is indexed, and the corresponding vector indexing type is clustering index.

[0136] If the selectivity of a node is less than the second threshold, the vector indexing method used by the node is determined to be no index.

[0137] In some possible implementations, the index determination module 402 is used to determine the selectivity of the node through the following steps:

[0138] The selectivity of a node is determined based on the ratio of the amount of business data corresponding to that node to the total amount of business data.

[0139] In some possible implementations, the data query device 400 further includes an index adjustment module 404, which is used for:

[0140] Upon receiving a vector index reconstruction request, the selectivity of each node in the multi-way search balanced search tree is redefined;

[0141] Adjust the vector indexing method used by the node based on the updated selectivity.

[0142] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0143] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0144] Based on the same technical concept, this application also provides a computer device 500, referring to... Figure 5 The diagram shown is a schematic representation of the structure of a computer device according to an exemplary embodiment of this application, comprising:

[0145] The processor 510, memory 520, and bus 530 are included. The memory 520 is used to store execution instructions and includes main memory 521 and external memory 522. The main memory 521, also known as internal memory, is used to temporarily store the operation data in the processor 510 and the data exchanged with external memory 522 such as hard disk. The processor 510 exchanges data with external memory 522 through main memory 521.

[0146] In this embodiment, the memory 520 is specifically used to store application code that executes the solution of this application, and its execution is controlled by the processor 510. That is, when the electronic device 500 is running, the processor 510 communicates with the memory 520 through the bus 530, or the processor 510 communicates with the memory 520 through other means, so that the processor 510 executes the application code stored in the memory 520, and then executes the steps of the data query method described in any of the foregoing embodiments.

[0147] The memory 520 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0148] Processor 510 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0149] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 500. In other embodiments of this application, the electronic device 500 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0150] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the data query method described in the above method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0151] This disclosure also provides a computer program product, which stores a computer program. When the computer program is run by a processor, it executes the steps of the data query method provided in any of the above embodiments of this disclosure. For details, please refer to the above method embodiments, which will not be repeated here.

[0152] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium, which can be a volatile or non-volatile computer-readable storage medium. In another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0153] Furthermore, embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0154] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0155] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0156] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0157] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0158] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0159] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0160] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data query method, characterized in that, The method includes: Based on multiple business data to be queried, a multi-way search balanced search tree is constructed according to at least one relational attribute; in the multi-way search balanced search tree, each non-leaf node is used to indicate the corresponding attribute range, and each leaf node is used to store the business data belonging to the attribute range indicated by the parent node of that leaf node. For each node of the multi-way search balanced search tree, a vector indexing method is determined for the node based on the node's selectivity; the selectivity indicates the frequency with which the node is accessed during the query process; the vector indexing method indicates whether there is an index or not, and the type of vector index when there is an index. In response to a query request, target business data is selected from the multiple business data based on the query attribute range and query vector corresponding to the query request, the attribute range corresponding to each non-leaf node of the multi-way search balanced search tree, and the vector indexing method corresponding to each node.

2. The method according to claim 1, characterized in that, The step of filtering target business data from the multiple business data based on the query attribute range and query vector corresponding to the query request, the attribute range corresponding to each non-leaf node of the multi-way search balanced search tree, and the vector indexing method corresponding to each node includes: Based on the query attribute range corresponding to the query request and the attribute range corresponding to each non-leaf node of the multi-way search balanced search tree, at least one target node is selected from the multi-way search balanced search tree. Based on the query vector corresponding to the query request, the business data corresponding to each target node is queried according to the vector indexing method corresponding to each target node to determine the target business data.

3. The method according to claim 2, characterized in that, The step of querying the business data corresponding to each target node based on the query vector corresponding to the query request and according to the vector indexing method corresponding to each target node to determine the target business data includes: For each target node, based on the query vector corresponding to the query request, the business data corresponding to the target node is queried according to the vector indexing method corresponding to the target node, and a first preset number of candidate business data with the highest similarity to the query vector are determined. The candidate service data with the highest similarity among the candidate service data corresponding to each target node are determined as the target service data.

4. The method according to claim 3, characterized in that, The step of querying the business data corresponding to the target node based on the query vector corresponding to the query request, according to the vector indexing method corresponding to the target node, and determining a first preset number of candidate business data with the highest similarity to the query vector, includes: If the vector indexing method corresponding to the target node is no index, a K-nearest neighbor query is performed on the business data corresponding to the target node based on the query vector corresponding to the query request to determine a first preset number of candidate business data with the highest similarity to the query vector. If the vector indexing method corresponding to the target node is indexed, based on the query vector corresponding to the query request, and according to the vector indexing type indicated by the vector indexing method, an approximate nearest neighbor query is performed on the business data corresponding to the target node to determine a first preset number of candidate business data with the highest similarity to the query vector.

5. The method according to claim 1, characterized in that, The step of determining the vector indexing method for a node based on its selectivity includes: If the selectivity of a node is greater than a first threshold, the node is determined to use an indexed vector indexing method, and the corresponding vector index type is a graph index. If the selectivity of a node is greater than or equal to the second threshold and less than or equal to the first threshold, it is determined that the vector indexing method used by the node is indexed, and the corresponding vector indexing type is clustering index. If the selectivity of a node is less than the second threshold, the vector indexing method used by the node is determined to be no index.

6. The method according to claim 1, characterized in that, The selectivity of the node is determined by the following steps: The selectivity of a node is determined based on the ratio of the amount of business data corresponding to that node to the total amount of business data.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: Upon receiving a vector index reconstruction request, the selectivity of each node in the multi-way search balanced search tree is redefined; Adjust the vector indexing method used by the node based on the updated selectivity.

8. A data query device, characterized in that, The device includes: The tree construction module is used to construct a multi-way search balanced search tree based on multiple business data to be queried, according to at least one relational attribute; in the multi-way search balanced search tree, each non-leaf node is used to indicate the corresponding attribute range, and each leaf node is used to store the business data belonging to the attribute range indicated by the parent node of that leaf node. The index determination module is used to determine the vector indexing method for each node of the multi-way search balanced search tree based on the selectivity of the node; the selectivity is used to indicate the frequency of the node being accessed during the query process; the vector indexing method is used to indicate whether there is an index or not, and the type of vector index when there is an index. The data filtering module is used to respond to a query request and filter out target business data from the multiple business data based on the query attribute range and query vector corresponding to the query request, the attribute range corresponding to each non-leaf node of the multi-way search balanced search tree, and the vector indexing method corresponding to each node.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the data query method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the data query method according to any one of claims 1 to 7.

Images