Data query method, device, electronic device and storage medium

Abstract

The present invention relates to the field of data query technology, and in particular to a data query method, device, electronic device, and storage medium. The method comprises: obtaining data to be queried, wherein the data to be queried includes structured conditions and vectors to be queried; customizing a joint query scanning path in a database management system; calculating the total cost of the joint query of the structured conditions and the vectors to be queried based on a cost model, selecting the path with the minimum total cost from the customized joint query scanning path as the target joint query scanning path and converting it into a target joint query scanning task, performing a joint query in the database management system based on the target joint query scanning task, and obtaining target query data corresponding to the data to be queried. The present application calculates the path with the minimum total cost as the target joint query scanning path through a cost model, and searches the target joint query scanning path in combination with the structured conditions to narrow the search range of the vector index, which is conducive to improving the query speed.

Description

Technical Field

[0001] The present invention relates to the field of data query technology, and in particular to a data query method, device, electronic device and storage medium. Background Art

[0002] In the existing technology, AIDB 2.x is often used to perform high-dimensional vector searches in PostgreSQL databases, meaning that PostgreSQL already supports vector searches. However, when processing joint query statements that combine structured conditions and vector searches, AIDB 2.x treats structured condition searches and vector searches as independent tasks. During execution, it first queries the vector index where the vector search is located, and then performs structured condition filtering on the vector search results. Because vector searches are generally time-consuming, and the vector search time in AIDB 2.x is proportional to the size of the underlying database, vector searches can take an excessively long time when the underlying database has a large amount of data. Therefore, the existing AIDB 2.x suffers from slow data search speeds. Summary of the Invention

[0003] An embodiment of the present invention provides a data query method, which aims to solve the problem of slow data search speed in the existing AIDB 2.x.

[0004] In a first aspect, an embodiment of the present invention provides a data query method, the method comprising the following steps:

[0005] Acquiring data to be queried, wherein the data to be queried includes structured conditions and a vector to be queried;

[0006] Customize the scan path for federated queries in the database management system;

[0007] Based on a preset cost model, the total cost of the joint query of the structured condition and the vector to be queried is calculated, and the path with the minimum total cost is selected from the custom joint query scan path as the target joint query scan path, wherein the target joint query scan path includes the query for the structured condition and the query for the vector;

[0008] Converting the target joint query scanning path into a target joint query scanning task;

[0009] Based on the target joint query scanning task, a joint query is performed in the database management system to obtain target query data corresponding to the data to be queried.

[0010] In a second aspect, an embodiment of the present invention provides a data query device, comprising:

[0011] An acquisition module, configured to acquire data to be queried, wherein the data to be queried includes structured conditions and a vector to be queried;

[0012] Custom modules for customizing the scan path of federated queries in database management systems;

[0013] a calculation module, configured to calculate the total cost of a joint query of the structured condition and the vector to be queried based on a preset cost model, and select a path with the minimum total cost from the custom joint query scan paths as a target joint query scan path, wherein the target joint query scan path includes queries for structured conditions and queries for vectors;

[0014] A conversion module, configured to convert the target joint query scanning path into a target joint query scanning task;

[0015] The query module is configured to perform a joint query in the database management system based on the target joint query scanning task to obtain target query data corresponding to the data to be queried.

[0016] In a third aspect, an electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a data query method provided in an embodiment of the present invention when executing the computer program.

[0017] In a fourth aspect, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements the steps of a data query method provided by an embodiment of the present invention.

[0018] In an embodiment of the present invention, by obtaining the data to be queried, the data to be queried includes structured conditions and vectors to be queried; a joint query scanning path is customized in the database management system; based on a preset cost model, the total cost of the joint query of the structured conditions and the vector to be queried is calculated, and the path with the smallest total cost is selected from the customized joint query scanning path as the target joint query scanning path, wherein the target joint query scanning path includes queries on structured conditions and queries on vectors; the target joint query scanning path is converted into a target joint query scanning task; based on the target joint query scanning task, a joint query is performed in the database management system to obtain the target query data corresponding to the data to be queried. This application provides users with the ability to customize scanning paths based on the PostgreSQL kernel. For joint query statements containing structured conditions and vector searches, a joint query scanning path is provided to the PostgreSQL kernel. The total cost of the joint query of the structured conditions and the vector to be queried is first calculated through the cost model, and the path with the smallest total cost is selected as the target joint query scanning path. The structured conditions are combined to search in the target joint query scanning path to narrow the search range of the vector index, which is conducive to improving the query speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0021] Figure 2a This is a flow chart of a data query method provided by an embodiment of the present invention;

[0022] Figure 2b Schematic diagrams of two structures of joint query provided by embodiments of the present invention;

[0023] Figure 3a is a flow chart of another data query method provided by an embodiment of the present invention;

[0024] Figure 3b This is a flow chart of a custom joint query scanning path provided by an embodiment of the present invention;

[0025] Figure 3c The embodiment of the present invention provides Figure 3c for Figure 2aFlowchart of step S210;

[0026] Figure 4 This is a structural diagram of a data query device provided by an embodiment of the present invention;

[0027] Figure 5 The embodiment of the present invention provides Figure 4 A schematic diagram of the structure of the custom module 402;

[0028] Figure 6 The embodiment of the present invention provides Figure 4 A schematic diagram of the structure of the calculation module 403;

[0029] Figure 7 The embodiment of the present invention provides Figure 4 A schematic structural diagram of the conversion module 404;

[0030] Figure 8 The embodiment of the present invention provides Figure 4 A schematic diagram of the structure of the query module 405;

[0031] Figure 9 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0033] The terms "including" and "having" and any variations thereof in the specification, claims and drawings of this application are intended to cover non-exclusive inclusions. The terms "first", "second" and the like in the specification, claims or drawings of this application are used to distinguish different objects rather than to describe a specific order. Reference to "embodiments" herein means that the specific features, structures or characteristics described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0034] like Figure 1As shown, system architecture 100 may include terminal devices 1001, 1002, and 1003, a network 1004, and a server 1005. Network 1004 is a medium for providing communication links between terminal devices 1001, 1002, and 1003 and server 1005. Network 1004 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0035] Users can use terminal devices 1001, 1002, and 1003 to interact with server 1005 via network 1004 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 1001, 1002, and 1003, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0036] Terminal devices 1001, 1002, and 1003 can be various electronic devices with display screens and support web browsing, including but not limited to smartphones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV), laptop computers, desktop computers, etc.

[0037] Server 1005 can be a server that provides various services, such as a backend server that supports the web pages displayed on terminal devices 1001, 1002, and 1003. In this application, a user can issue a query request through terminal devices 1001, 1002, and 1003, and the query request includes the data to be queried. A database management system can be used in server 1005, and server 1005 can respond to the query request issued by the user.

[0038] It should be noted that the data query method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the data query device is generally set in the server / terminal device.

[0039] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0040] See Figure 2a , Figure 2a This is a flow chart of a data query method provided by an embodiment of the present invention. Figure 1As shown in the figure, the method includes the following steps:

[0041] S101. Obtain the data to be queried, where the data to be queried includes a structured condition and a query vector.

[0042] In an embodiment of the present invention, the electronic device on which the above data query method runs can obtain the data to be queried and feedback the target query data through a wired connection or a wireless connection. Among them, the wireless connection method may include, but is not limited to, 3G / 4G connection, WiFi (Wireless-Fidelity) connection, Bluetooth connection, WiMAX (Worldwide Interoperability for Microwave Access) connection, Zigbee (low-power local area network protocol, also known as the Purple Peak protocol) connection, UWB (ultra wideband) connection, and other currently known or future-developed wireless connection methods.

[0043] Among them, the data to be queried may be data included in a query request issued by a user based on a mobile terminal, and the data may include a structured condition (Structured Scan) and a query vector (IVFPQ Scan). The structured condition may include classification attributes corresponding to the query vector. For example, it may include identity attributes (such as gender, name, age, ID number, etc.), appearance attributes (such as height, three dimensions, hat, glasses, clothes, pants, shoes worn, and style and color information of the hat, clothes, pants, shoes, etc.), behavior attributes (such as communication, walking, running, driving, etc.), and one or more of them. The above query vector may be a sorting code corresponding to the data to be queried, or may also be called a short feature value. The query statement including the structured condition and the search vector is as follows:

[0044] SELECT id FROM face

[0045] WHERE create_time>t1 AND create_time<t2--Structured condition

[0046] ORDER BY feature<=>query_feature--Vector search

[0047] LIMIT K;

[0048] S102. Customize a combined query scan path in the database management system.

[0049] Because it targets a federated query statement, a new scan type needs to be provided to the PostgreSQL (database management system) kernel, which means a customized federated query scan path is required. Of course, PostgreSQL can also include non-federated queries. When the acquired query data only includes the query vector without structured conditions, a non-federated query method can be used to query. Specifically, a customized federated query scan path can be generated during planning using the set_rel_pathlist_hook global variable. This is a federated query scan path for structured conditions and vector searches.

[0050] S103. Based on a preset cost model, calculate the total cost of the joint query of the structured condition and the vector to be queried, and select the path with the minimum total cost from the custom joint query scanning path as the target joint query scanning path, wherein the target joint query scanning path includes the query for the structured condition and the query for the vector.

[0051] In PostgreSQL, the query planner creates multiple query paths for each query request and then selects the path with the lowest cost to execute, which is the cost model mentioned above. The cost is divided into IO (In / Out) cost and CPU (Central Processing Unit) cost. The IO cost and CPU cost are related to the number of disk accesses and the amount of calculation performed during the scan, respectively. Based on the cost model, the total cost of the joint query of the structured condition and the query vector can be calculated in the multiple query paths created. The total cost is a value, and the minimum total cost is selected from the total costs corresponding to the multiple query paths. The query path corresponding to the minimum total cost is used as the target joint query scan path. Reference Figure 2b As shown in the figure, the left side is the non-joint query scanning path, and the right side is the target joint query scanning path. The Limit operation and the filter operation are implemented by the Limit operator and the filter operator in the PostgreSQL database. Figure 2b It can be seen that the non-joint query path is only applicable to vector indexes, while the target joint query scan path can combine structured and vector indexes for joint queries. The left child node of the target joint query scan path is a structured index, and the right child node is a vector index. The vector index used in this embodiment can be the IVFPQ algorithm. Structured conditional queries and vector queries can then be performed based on the target joint query scan path. Calculating the optimal path using a cost model can reduce the time and computational complexity of subsequent queries.

[0052] S104: Convert the target joint query scanning path into a target joint query scanning task.

[0053] Among them, after obtaining the target joint query scanning path, it can be converted into a target joint query scanning task (target joint query scanning plan) through a function. Specifically, the target joint query scanning path can be converted into a target joint query scanning plan through the callback in the methods of the CustomPath function.

[0054] S105 : Perform a joint query in the database management system based on the target joint query scanning task to obtain target query data corresponding to the data to be queried.

[0055] After converting the target joint query scan path into a target joint query scan plan, the target joint query scan plan can be executed to perform a joint query in PostgreSQL to obtain target query data. The target query data is the query result for the target query data and can be a feature vector corresponding to the target query vector.

[0056] In an embodiment of the present invention, by obtaining the data to be queried, the data to be queried includes structured conditions and vectors to be queried; a joint query scanning path is customized in the database management system; based on a preset cost model, the total cost of the joint query of the structured conditions and the vector to be queried is calculated, and the path with the smallest total cost is selected from the customized joint query scanning path as the target joint query scanning path, wherein the target joint query scanning path includes queries on structured conditions and queries on vectors; the target joint query scanning path is converted into a target joint query scanning task, and a joint query is performed in the database management system based on the target joint query scanning task to obtain target query data corresponding to the data to be queried. This application is based on the ability of the PostgreSQL kernel to provide users with a customized scanning path. For joint query statements containing structured conditions and vector searches, a joint query scanning path is provided to the PostgreSQL kernel. The total cost of the joint query of the structured conditions and the vector to be queried is first calculated through the cost model, and the path with the smallest total cost is selected as the target joint query scanning path. The structured conditions are combined to search in the target joint query scanning path to narrow the search range of the vector index, which is conducive to improving the query speed.

[0057] See Figure 3a , Figure 3a This is a flow chart of another data query method provided by an embodiment of the present invention. Figure 3a As shown, the steps include:

[0058] S201: Acquire data to be queried, where the data to be queried includes structured conditions and a vector to be queried.

[0059] S202: Determine whether to query from the original table in the database management system.

[0060] S203: If the query is performed from the original table in the database management system, it is determined whether the data to be queried contains a structured condition, and a structured index is created for the column where the structured condition is located.

[0061] S204: If the data to be queried contains a structured condition, and a structured index is created for the column containing the structured condition, determine whether the data to be queried contains a vector to be queried, and whether a vector index is created for the column containing the vector to be queried.

[0062] S205: If the data to be queried contains the vector to be queried, and a vector index is created for the column containing the vector to be queried, all structured indexes and vector indexes are searched, and scanning paths corresponding to the structured indexes and scanning paths corresponding to the vector indexes are created respectively.

[0063] PostgreSQL provides the set_rel_pathlist_hook global variable to add scan paths to the kernel when generating custom federated query scan paths during planning. The definition of the set_rel_pathlist_hook variable is as follows:

[0064] / *Hook for plugins to get control in set_rel_pathlist()* /

[0065] Typedef void(*set_rel_pathlist_hook_type)(Planner Info*root,

[0066] ReloptInfo*rel,

[0067] Index rti,

[0068] RangeTblEntry*rte);

[0069] extern PGDLLIMPORT set_rel_pathlist_hook_type set_rel_pathlist_hook;

[0070] The set_rel_pathlist_hook variable points to a user-defined function, in which a CustomPath object needs to be defined and initialized. Its definition is as follows:

[0071]

[0072]

[0073] Among them, path contains the joint query scan path composed of structured index and vector index. The initialization of path member is the initialization of joint query scan path. Figure 3b As shown, Figure 3b The flowchart of the custom joint query scanning path provided by the embodiment of the present invention has the following main processes:

[0074] First, the conditions for the joint query need to be determined. Only when the conditions are met can the joint query scan path be created. There are three conditions that need to be met for a joint query: a. Determine whether the query is to be performed from the original table in the database management system; b. Determine whether the data to be queried contains structured conditions, and whether a structured index (btree index) has been created for the column containing the structured conditions; c. Determine whether the data to be queried contains the vector to be queried, and whether a vector index has been created for the column containing the vector to be queried. A joint query can only be performed when all three conditions are met simultaneously; if any one of them is not met, a joint query cannot be performed.

[0075] When the above three conditions are met at the same time, all structured indexes and vector indexes can be searched, and corresponding scan paths can be created according to all structured indexes, and corresponding scan paths can be created according to vector indexes.

[0076] S206: Create a joint query scan path based on the scan path of the structured index and the scan path of the vector index.

[0077] After respectively obtaining the scan path based on the structured index and the scan path based on the vector index, the two can be combined to obtain multiple combined query scan paths.

[0078] S207. Calculate the total cost of the path for a joint query involving the structured condition and the vector to be queried based on the cost of executing the structured index, the number of rows in the original table, the ratio of the query results for the structured condition to the number of rows in the table, the cost of obtaining the index position of the row, the cost of obtaining the index page, and the cost of calculating the distance between the data to be queried and the data in the vector index.

[0079] The specific calculation formula of the cost model is as follows:

[0080] C hybrid =c structured +r1*n total *(c0+c1+c2) (1)

[0081] C not_hybrid =(r2*n total ) / n1*c1+r2*n*c2 (2)

[0082] Among them, C hybrid is the total cost of the joint query, C not_hybrid is the cost of non-joint query, c structured is the cost of executing a btree index, r1 is the ratio of query results for structured conditions to the number of table rows, ntotal is the number of table rows, c0 is the cost of retrieving the row's index position from the hash table, c1 is the I / O cost of retrieving the index page, c2 is the cost of calculating the distance between the query data and the data in the vector index, r2 is the ratio of the number of vector index searches to the number of table rows for non-join queries, and n1 stores the number of index elements for each index page. The hash table above stores the mapping between row positions and index element positions.

[0083] The above formula (1) is the cost model of joint query, and formula (2) is the cost model of non-joint query. Substituting the above parameters into formula (1) and formula (2) can calculate the total cost of joint query and the total cost of non-joint query. When the cost of joint query C hybrid Less than the cost of non-join query C not_hybrid , the PostgreSQL planner automatically chooses to perform a union query.

[0084] S208 : Selecting a path with the minimum total cost from the customized joint query scanning paths as a target joint query scanning path.

[0085] During the query process, the PostgreSQL planner automatically selects a joint query and selects the path with the smallest total cost from multiple calculated total costs as the target joint query scan path to query structured conditions and vectors.

[0086] S209: Convert the target joint query scanning path into a target joint query scanning task.

[0087] S210 : Perform a joint query in the database management system based on the target joint query scanning task to obtain target query data corresponding to the data to be queried.

[0088] Optionally, the above step S209 specifically includes:

[0089] Generates a scan task for the structured index through the structured index of the target joint query scan path.

[0090] Generate a vector index scan task by scanning the vector index of the target joint query scan path.

[0091] Combine the structured index scanning task and the vector index scanning task to generate the target joint query scanning task.

[0092] Specifically, the target joint query scan path can be converted into a target joint query scan task based on a conversion function. The conversion function can refer to the callback in the methods of the CustomPath object. The methods member is used for the target joint query scan path callback to convert the target joint query scan path into a target joint query scan path plan. Finally, the CustomPath object needs to be added to the query path through the add_path function. At the same time, this callback will also generate a CustomScanState object for the CustomScan object to execute the custom query plan. The target joint query scan path is represented by the CustomScan object, which is defined as follows:

[0093]

[0094] The scan function is obtained according to the path in the CustomPath object, and the initialization of the scan member is the initialization of the target joint query scan plan.

[0095] Combine Figure 2b As shown in the figure, we first obtain the structured index scan task by scanning the left child node (structured index) of the target joint query scan path. Then, we obtain the vector index scan task by scanning the right child node (vector index) of the joint query scan path. After obtaining the structured index scan task and the vector index scan task, we combine them to obtain the joint query scan plan.

[0096] Optional, reference Figure 3c As shown, Figure 3c for Figure 2a Flowchart of step S210 in FIG. The above step S210 includes:

[0097] S2101. Based on a preset query function, search for query results that meet the structured conditional index, where the query results include row positions of data that meet the structured conditional index.

[0098] Specifically, when executing a target joint query scan task, PostgreSQL saves the execution status through CustomScanState or its derived class. Its definition is as follows:

[0099]

[0100]

[0101] Among them, ss stores the status of the target joint query scan task during execution, and methods points to a CustomExecMethods object, which is used to manage the execution process of the custom scan plan. Its definition is as follows:

[0102]

[0103]

[0104] The preset query function may be a BeginCustomScan member function, which is used to start a joint query, and mainly initialize the query status of the query node and its child nodes.

[0105] The above ExecCustomScan member function is used to execute a union query.

[0106] The EndCustomScan member function is used to end the scan after completion. In the present invention, the EndCustomScan member function executes the ExecEndScan function on the left child node (structured index) and right child node (vector index) of the target joint query scan plan. The ExecEndScan function automatically cleans up and releases the incoming query nodes, thereby ending the target joint query scan plan.

[0107] The ReScanCustomScan member function is used to restart a scan. In the present invention, the ReScanCustomScan member function executes the ExecReScan function on the left child node (structured index) and right child node (vector index) of the target joint query scan plan. The ExecReScan function automatically resets the state of the passed query node, allowing the new joint query scan plan to be re-executed.

[0108] The ExplainCustomScan command encapsulates and outputs the target query plan information for use in the Explain command's query plan display.

[0109] In the present invention, the target joint query scan plan narrows the search range of the vector index through the structured conditions. Therefore, the query (IndexScan) of the structured index is first executed through the ExecCustomScan member function to obtain the row positions of all data that meet the structured conditions.

[0110] S2102. Pass the query result into the vector index for query, filter the row position where the data that meets the vector index is located, obtain the index element position corresponding to the row position from the built-in data position mapping table, and read the feature vector of the index element at the index element position.

[0111] Among them, the row positions of all data that meet the structured conditions obtained in the query results can be passed into the query conditions of the vector index. When executing the vector index query (IndexScan), the index element position will be obtained from the hash table (data position mapping table) that stores the mapping relationship between the row position and the index element position based on the row positions of all data that meet the structured conditions. The feature vector of the index element is read from the corresponding index element position, and multiple feature vectors will be obtained in the end.

[0112] S2103: Calculate the distance between the feature vector of the index element at the index element position and the vector to be queried, and use the feature vector that meets the preset distance as target query data.

[0113] Among them, the above-mentioned multiple feature vectors are obtained, and the distance between each feature vector and the vector to be queried can be calculated. When the traversal of the feature vectors corresponding to the row positions in the query results of the structured conditions is completed, the entire joint query is completed, and the calculated multiple distances can be compared with the preset distances, and the feature vectors that meet the preset distances can be output as target query data.

[0114] In an embodiment of the present invention, based on the user-defined scan path capability provided by the PostgreSQL kernel, a joint query scan path is provided to the PostgreSQL kernel for joint query statements containing structured conditions and vector searches. A cost model is used to first calculate the total cost of the joint query of the structured conditions and the vector to be queried, and the path with the lowest total cost is selected as the target joint query scan path. This saves time in data path selection and accelerates the query process. Furthermore, by first performing a query based on the structured conditions, then finding the feature vectors corresponding to all row positions in the query results for the structured conditions, calculating their distances to the vector to be queried, and selecting target query data based on the calculated distances, the search range of the vector index can be narrowed, which helps improve query speed.

[0115] See Figure 4 , Figure 4 is a structural diagram of a data query device provided by an embodiment of the present invention, such as Figure 4 As shown, the apparatus 400 includes:

[0116] An acquisition module 401 is used to acquire data to be queried, where the data to be queried includes structured conditions and a vector to be queried;

[0117] A customization module 402 is used to customize a joint query scanning path in a database management system;

[0118] Calculation module 403, configured to calculate the total cost of a joint query of the structured condition and the query vector based on a preset cost model, and select the path with the minimum total cost from the custom joint query scan paths as a target joint query scan path, wherein the target joint query scan path includes queries for the structured condition and queries for the vector;

[0119] A conversion module 404 is configured to convert a target joint query scanning path into a target joint query scanning task;

[0120] The query module 405 is configured to perform a joint query in the database management system based on the target joint query scanning task to obtain target query data corresponding to the data to be queried.

[0121] Optionally, the custom module 402 is further configured to initialize the scan path in the database management system through preset global variables and create a joint query scan path.

[0122] Optional, such as Figure 5 As shown, Figure 5 The embodiment of the present invention provides Figure 4 Schematic diagram of the structure of the custom module, wherein the custom module 402 includes:

[0123] The first determination unit 4021 is used to determine whether to query from the original table in the database management system;

[0124] The second judgment unit 4022 is used to judge whether the data to be queried contains a structured condition and a structured index is created for the column where the structured condition is located, if the query is performed from the original table in the database management system;

[0125] The third judgment unit 4023 is configured to, if the data to be queried contains a structured condition and a structured index is created for the column containing the structured condition, determine whether the data to be queried contains a vector to be queried and whether a vector index is created for the column containing the vector to be queried;

[0126] A first search unit 4024 is configured to, if the query data includes a query vector and a vector index is created for the column containing the query vector, search all structured indexes and vector indexes, and create a scan path corresponding to the structured index and a scan path corresponding to the vector index, respectively.

[0127] The creating unit 4025 is configured to create a joint query scanning path based on the scanning path of the structured index and the scanning path of the vector index.

[0128] Optional, such as Figure 6 As shown, Figure 6 The embodiment of the present invention provides Figure 4 Schematic diagram of the structure of the calculation module, wherein the calculation module 403 includes:

[0129] A first calculation unit 4031 is configured to calculate a total cost of a path for a joint query involving the structured condition and the query vector based on the cost of executing the structured index, the number of rows in the original table, the ratio of query results for the structured condition to the number of rows in the table, the cost of obtaining the index position of the row, the cost of obtaining the index page, and the cost of calculating the distance between the query data and the data in the vector index.

[0130] The selection unit 4032 is configured to select a path with the minimum total cost from the user-defined joint query scanning paths as a target joint query scanning path.

[0131] Optional, such as Figure 7 As shown, Figure 7 The embodiment of the present invention provides Figure 4 Schematic diagram of the structure of the conversion module, wherein the conversion module 404 includes:

[0132] A first generating unit 4041 is configured to generate a structured index scanning task using the structured index of the target joint query scanning path;

[0133] The second generating unit 4042 is configured to generate a vector index scanning task using the vector index of the target joint query scanning path;

[0134] The third generating unit 4043 is configured to combine the structured index scanning task and the vector index scanning task to generate a target joint query scanning task.

[0135] like Figure 8 As shown, Figure 8 The embodiment of the present invention provides Figure 4 Schematic diagram of the structure of the query module, wherein the query module 405 includes:

[0136] The second search unit 4051 is configured to search for a query result that satisfies the structured conditional index based on a preset query function, wherein the query result includes a row position of data that satisfies the structured conditional index;

[0137] The filtering unit 4052 is used to pass the query result into the vector index for query, filter the row position where the data that meets the vector index is located, obtain the index element position corresponding to the row position from the built-in data position mapping table, and read the feature vector of the index element at the index element position;

[0138] The second calculation unit 4053 is configured to calculate the distance between the feature vector of the index element at the index element position and the query vector, and use the feature vector that meets the preset distance as the target query data.

[0139] The data query device provided by the embodiment of the present invention can realize Figure 1 To avoid repetition, the various implementation methods and corresponding beneficial effects in the data query method embodiment in -3 are not described here.

[0140] like Figure 9 As shown, Figure 9 This is a structural diagram of an electronic device provided by an embodiment of the present invention. Figure 9 As shown, it includes: a memory 902, a processor 901, a network interface 903, and a computer program stored in the memory 902 and executable on the processor 901, wherein:

[0141] The processor 901 is configured to call the computer program stored in the memory 902 and execute the following steps:

[0142] Obtaining the data to be queried, which includes structured conditions and the query vector;

[0143] Customize the scan path for federated queries in the database management system;

[0144] Based on a preset cost model, the total cost of the joint query of the structured condition and the query vector is calculated, and the path with the minimum total cost is selected from the custom joint query scan path as the target joint query scan path, where the target joint query scan path includes the query for the structured condition and the query for the vector;

[0145] Converting the target joint query scanning path into a target joint query scanning task;

[0146] Based on the target joint query scanning task, a joint query is performed in the database management system to obtain target query data corresponding to the data to be queried.

[0147] Optionally, the processor 901 customizes the joint query scan path in the database management system, including:

[0148] The scanning path is initialized in the database management system through preset global variables, and a joint query scanning path is created.

[0149] Optionally, the processor 901 initializes the scan path in the database management system through preset global variables and creates a joint query scan path, including:

[0150] Determine whether to query from the original table in the database management system;

[0151] If the query is from the original table in the database management system, it is determined whether the data to be queried contains structured conditions and whether a structured index is created for the column where the structured conditions are located;

[0152] If the data to be queried contains structured conditions, and a structured index is created for the column containing the structured conditions, then determine whether the data to be queried contains the query vector, and whether a vector index is created for the column containing the query vector.

[0153] If the query data contains the query vector, and a vector index is created for the column containing the query vector, all structured indexes and vector indexes are searched, and scan paths corresponding to the structured indexes and vector indexes are created respectively.

[0154] A joint query scan path is created based on the scan path of the structured index and the scan path of the vector index.

[0155] Optionally, the processor 901 calculates the total cost of the path when jointly querying the structured condition and the query vector based on a preset cost model, and selects the path with the minimum total cost from the custom joint query scan paths as the target joint query scan path, including:

[0156] The total cost of the path for a joint query involving the structured condition and the query vector is calculated based on the cost of executing the structured index, the number of rows in the original table, the proportion of the query results for the structured condition to the total number of rows in the table, the cost of obtaining the index location of the row, the cost of obtaining the index page, and the cost of calculating the distance between the query data and the data in the vector index.

[0157] The path with the minimum total cost is selected from the customized joint query scanning paths as the target joint query scanning path.

[0158] Optionally, the processor 901 converts the target joint query scan path into a target joint query scan task, including:

[0159] Generate a structured index scan task from the structured index of the target joint query scan path;

[0160] Generate a vector index scan task by using the vector index of the target joint query scan path;

[0161] Combine the structured index scanning task and the vector index scanning task to generate the target joint query scanning task.

[0162] Optionally, the target-based joint query scanning task executed by the processor 901 performs a joint query in the database management system to obtain target query data corresponding to the data to be queried, including:

[0163] Based on the preset query function, search for query results that meet the structured condition index, and the query results include the row position of the data that meets the structured condition index;

[0164] Pass the query result to the vector index for query, filter the row position where the data that meets the vector index is located, obtain the index element position of the corresponding row position from the built-in data position mapping table, and read the feature vector of the index element at the index element position;

[0165] The distance between the feature vector of the index element at the index element position and the query vector is calculated, and the feature vector that meets the preset distance is used as the target query data.

[0166] The electronic device provided in the embodiment of the present invention can implement various implementation methods in the data query method embodiment and the corresponding beneficial effects, which will not be described again here to avoid repetition.

[0167] It should be noted that the figure only shows 901-903 with components, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art will understand that the electronic device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0168] The electronic device 900 may be a computing device such as a desktop computer, a notebook computer, a PDA, a cloud server, etc. The electronic device 900 may interact with the user through a keyboard, a mouse, a remote control, a touchpad, or a voice control device.

[0169] The memory 902 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, the memory 901 can be an internal storage unit of the electronic device, such as the hard disk or memory of the electronic device. In other embodiments, the memory 901 can also be an external storage device of the electronic device, such as a plug-in hard disk equipped on the electronic device, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, etc. Of course, the memory 901 can also include both the internal storage unit of the electronic device and its external storage device. In this embodiment, the memory 901 is generally used to store the operating system and various application software installed on the electronic device, such as the program code of the data query method. In addition, the memory 901 can also be used to temporarily store various types of data that have been output or are about to be output.

[0170] In some embodiments, the processor 901 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 902 is generally used to control the overall operation of the electronic device. In this embodiment, the processor 901 is used to execute program code stored in the memory 901 or process data, such as executing the program code of the data query method.

[0171] The network interface 903 may include a wireless network interface or a wired network interface. The network interface 903 is generally used to establish a communication connection between the electronic device 900 and other electronic devices.

[0172] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by the processor 901, the various processes of the data query method embodiment provided by the embodiment of the present invention are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0173] Those skilled in the art will appreciate that all or part of the processes in the data query method of the embodiment can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes of the embodiments of the method. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory 902 (RAM).

[0174] The above disclosure is merely a preferred embodiment of the present invention and certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A data query method, characterized in that: The method comprises the following steps: Acquiring data to be queried, wherein the data to be queried includes structured conditions and a vector to be queried; Customize the scan path for federated queries in the database management system; Based on a preset cost model, the total cost of the joint query of the structured condition and the vector to be queried is calculated, and the path with the minimum total cost is selected from the custom joint query scan path as the target joint query scan path, wherein the target joint query scan path includes the query for the structured condition and the query for the vector; Converting the target joint query scanning path into a target joint query scanning task; Performing a joint query in the database management system based on the target joint query scanning task to obtain target query data corresponding to the data to be queried; The customizing the joint query scanning path in the database management system includes: Initializing the scan path in the database management system through preset global variables and creating the joint query scan path; Initializing the scan path in the database management system through the preset global variables and creating the joint query scan path includes: Determining whether to query from an original table in the database management system; If the query is performed from the original table in the database management system, it is determined whether the data to be queried contains the structured condition and a structured index is created for the column where the structured condition is located; If the data to be queried contains the structured condition, and a structured index is created for the column where the structured condition is located, then determining whether the data to be queried contains a vector to be queried, and whether a vector index is created for the column where the vector to be queried is located; If the data to be queried contains a vector to be queried, and a vector index is created for the column containing the vector to be queried, all the structured indexes and the vector indexes are searched, and a scan path corresponding to the structured index and a scan path corresponding to the vector index are created respectively; Creating the joint query scan path based on the scan path of the structured index and the scan path of the vector index; Calculating the total cost of the joint query of the structured condition and the query vector based on a preset cost model, and selecting the path with the minimum total cost from the custom joint query scan paths as the target joint query scan path, including: Calculating the total cost of a path for a joint query involving the structured condition and the vector to be queried based on the cost of executing the structured index, the number of rows in the original table, the proportion of query results of the structured condition to the number of rows in the table, the cost of obtaining the index position of the row, the cost of obtaining the index page, and the cost of calculating the distance between the data to be queried and the data in the vector index; Selecting the path with the minimum total cost from the custom joint query scanning paths as the target joint query scanning path; The specific calculation formula of the cost model is as follows: （1） （2） Among them, C hybrid is the total cost of the joint query, C not_hybrid is the cost of non-joint query, c structured is the cost of executing the btree index, r1 is the ratio of the query results of the structured condition to the number of table rows, n total is the number of table rows, c0 is the cost of getting the index position of the row from the hash table, c1 is the IO cost of getting the index page, c2 is the cost of calculating the distance between the query data and the data in the vector index, r2 is the ratio of the number of vector index searches to the number of table rows in non-joint queries, and n1 is the number of index elements stored for each index page; When the cost of the joint query C hybrid Less than the cost of non-join query C not_hybrid , select Perform a joint query.

2. The method according to claim 1, wherein The converting the target joint query scanning path into a target joint query scanning task includes: Generate a scan task of the structured index by using the structured index of the target joint query scan path; Generate a scan task of the vector index by querying the vector index of the scan path through the target joint query; The scanning task of the structured index and the scanning task of the vector index are combined to generate the target joint query scanning task.

3. The method according to claim 1, wherein The target-based joint query scanning task performs a joint query in the database management system to obtain target query data corresponding to the data to be queried, including: Based on a preset query function, searching for a query result that satisfies the structured condition index, wherein the query result includes a row position of data that satisfies the structured condition index; Passing the query result into the vector index for query, filtering the row position where the data that meets the vector index is located, obtaining the index element position corresponding to the row position from the built-in data position mapping table, and reading the feature vector of the index element at the index element position; The distance between the feature vector of the index element at the index element position and the vector to be queried is calculated, and the feature vector that meets the preset distance is used as the target query data.

4. A data query device, characterized in that: include: An acquisition module, configured to acquire data to be queried, wherein the data to be queried includes structured conditions and a vector to be queried; Custom modules for customizing the scan path of federated queries in database management systems; a calculation module, configured to calculate the total cost of a joint query of the structured condition and the vector to be queried based on a preset cost model, and select a path with the minimum total cost from the custom joint query scan paths as a target joint query scan path, wherein the target joint query scan path includes queries for structured conditions and queries for vectors; A conversion module, configured to convert the target joint query scanning path into a target joint query scanning task; A query module, configured to perform a joint query in the database management system based on the target joint query scanning task to obtain target query data corresponding to the data to be queried; The custom module is further used to initialize the scan path in the database management system through preset global variables and create the joint query scan path; The custom module is also used to: Determining whether to query from an original table in the database management system; If the query is performed from the original table in the database management system, it is determined whether the data to be queried contains the structured condition and a structured index is created for the column where the structured condition is located; If the data to be queried contains the structured condition, and a structured index is created for the column where the structured condition is located, then determining whether the data to be queried contains a vector to be queried, and whether a vector index is created for the column where the vector to be queried is located; If the data to be queried contains a vector to be queried, and a vector index is created for the column containing the vector to be queried, all the structured indexes and the vector indexes are searched, and a scan path corresponding to the structured index and a scan path corresponding to the vector index are created respectively; Creating the joint query scan path based on the scan path of the structured index and the scan path of the vector index; The calculation module is specifically used for: Calculating the total cost of a path for a joint query involving the structured condition and the vector to be queried based on the cost of executing the structured index, the number of rows in the original table, the proportion of query results of the structured condition to the number of rows in the table, the cost of obtaining the index position of the row, the cost of obtaining the index page, and the cost of calculating the distance between the data to be queried and the data in the vector index; Selecting the path with the minimum total cost from the custom joint query scanning paths as the target joint query scanning path; The specific calculation formula of the cost model is as follows: （1） （2） Among them, C hybrid is the total cost of the joint query, C not_hybrid is the cost of non-joint query, c structured is the cost of executing the btree index, r1 is the ratio of the query results of the structured condition to the number of table rows, n total is the number of table rows, c0 is the cost of getting the index position of the row from the hash table, c1 is the IO cost of getting the index page, c2 is the cost of calculating the distance between the query data and the data in the vector index, r2 is the ratio of the number of vector index searches to the number of table rows in non-joint queries, and n1 is the number of index elements stored for each index page; When the cost of the joint query C hybrid Less than the cost of non-join query C not_hybrid , select Perform a joint query.

5. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a data query method according to any one of claims 1 to 3 when executing the computer program.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the data query method according to any one of claims 1 to 3 are implemented.

Images