A data query method and device, computer device and program product

Abstract

The present disclosure provides a data query method and device, computer equipment and program product, wherein the method comprises: analyzing a query request input by a user to determine vector query content and structured query language (SQL) content corresponding to the query request; performing vector index query according to the vector query content and a vector index column in a pre-constructed virtual table to determine a vector query result; the virtual table is used for data mapping of at least one physical table and comprises a vector index column corresponding to table data under at least one target column in the physical table; the vector index column is used for storing vector indexes of the table data under the target column and table information of a physical table to which the table data belongs; performing federated retrieval on the physical table according to the SQL content, the vector query result and a mapping relationship between the virtual table and the physical table to obtain a data query result.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data query method, apparatus, computer equipment, and program product. Background Art

[0002] Federated search, a method for performing cross-data source queries on multiple data sources, can significantly improve query efficiency in distributed databases, data lake architectures, and cross-system data analysis scenarios. Vector index query, a method for querying data using data vectors, can also significantly improve query efficiency for complex, unstructured data such as images, text, audio, and video.

[0003] If federated search and vector index query can be combined, data query efficiency will be further improved. Therefore, how to achieve the combined use of federated search and vector index query has become a technical difficulty worthy of attention. Summary of the Invention

[0004] The embodiments of the present disclosure at least provide a data query method, apparatus, computer device, and program product.

[0005] In a first aspect, an embodiment of the present disclosure provides a data query method, comprising:

[0006] Parsing the query request input by the user to determine the vector query content and structured query language SQL content corresponding to the query request;

[0007] Performing a vector index query based on the vector query content and a vector index column in a pre-constructed virtual table to determine a vector query result; the virtual table is used to perform data mapping on at least one physical table and includes a vector index column corresponding to table data under at least one target column in the physical table; the vector index column is used to store the vector index of the table data under the target column and table information of the physical table to which the table data belongs;

[0008] According to the SQL content, the vector query result, and the mapping relationship between the virtual table and the physical table, a federated search is performed on the physical table to obtain a data query result.

[0009] In a possible implementation, performing a vector index query based on the vector query content and a vector index column in a pre-built virtual table to determine a vector query result includes:

[0010] Performing a vector index query based on the vector query content and each vector index in the vector index column to obtain a similarity between each vector index and the vector query content;

[0011] filtering out a plurality of target vector indexes from the vector indexes based on the similarity, and obtaining target table information corresponding to each of the target vector indexes from the virtual table; the target table information includes a target physical table where target table data associated with the target vector index is located and a position in the target physical table.

[0012] In a possible implementation, the vector index is associated with a plurality of data vectors corresponding to table data.

[0013] The filtering out of the plurality of target vector indexes from the vector indexes based on the similarity, and the obtaining of the target table information corresponding to each of the target vector indexes from the virtual table include:

[0014] sorting each vector index in descending order of similarity, and regarding a vector index with a sorting order smaller than a target order as a target vector index;

[0015] filtering out a target data vector from each data vector associated with the target vector index based on similarity between the data vector and the vector query content;

[0016] obtaining target table information of target table data corresponding to each target data vector from the virtual table.

[0017] In a possible implementation, the federated retrieval of the physical table based on the SQL content, the vector query result, and the mapping relationship between the virtual table and the physical table to obtain a data query result includes:

[0018] determining each target physical table related to the query request according to each target table information in the vector query result and the mapping relationship;

[0019] generating a target query statement matching a syntax format of each target physical table according to the SQL content and a target vector index in the vector query result;

[0020] performing federated retrieval on each target physical table by using the target query statement corresponding to each target physical table to obtain a data query result.

[0021] In a possible implementation, the virtual table is constructed according to the following steps:

[0022] determining a vector index corresponding to table data of at least one target column and table information of a physical table corresponding to the vector index according to a data vector corresponding to the table data of the at least one target column in a plurality of physical tables to be mapped;

[0023] A virtual table for mapping the multiple physical tables is constructed according to the data attribute information of the multiple physical tables to be mapped and a vector index column composed of the vector index and the table information.

[0024] In a possible implementation, after constructing the virtual table, the method further includes:

[0025] In response to inserting new table data into the target column in the physical table, the vector index column in the constructed virtual table is updated according to the data vector corresponding to the new table data.

[0026] In a possible implementation, after obtaining the data query result, the method further includes:

[0027] Utilize the artificial intelligence model to generate query feedback information based on the data query results.

[0028] In a second aspect, an embodiment of the present disclosure further provides a data query device, comprising:

[0029] A parsing module, configured to parse a query request input by a user and determine the vector query content and structured query language SQL content corresponding to the query request;

[0030] a determination module configured to perform a vector index query based on the vector query content and a vector index column in a pre-constructed virtual table to determine a vector query result; the virtual table being configured to perform data mapping on at least one physical table and including a vector index column corresponding to table data under at least one target column in the physical table; the vector index column being configured to store a vector index of the table data under the target column and table information of the physical table to which the table data belongs;

[0031] A retrieval module is used to perform a federated search on the physical table according to the SQL content, the vector query result, and the mapping relationship between the virtual table and the physical table to obtain a data query result.

[0032] In a third aspect, an optional implementation of the present disclosure further provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are performed.

[0033] In a fourth aspect, an optional implementation of the present disclosure further provides a computer program product, including a computer program, which, when executed, implements the above-mentioned first aspect, or the steps in any possible implementation of the first aspect.

[0034] The data query method, apparatus, computer device, and program product provided by the embodiments of the present disclosure pre-construct a virtual table for data mapping to a physical table, and set a vector index column for the table data in the target column of the physical table in the virtual table. This allows the target vector index in the virtual table to be located according to the vector index column during the query, thereby implementing a vector index query. Based on the mapping relationship between the target vector index in the virtual table and the physical table, a federated search of the physical table can be implemented, thereby improving the efficiency of determining the query results. In this way, based on the virtual table and the vector index column in the virtual table, an efficient combination of federated search and vector index query can be achieved. Furthermore, by introducing a query request with vector query content, the vector query content for the vector index query and the SQL content for the federated search can be obtained based on the automatic parsing of the query request input by the user once, and the corresponding vector index query and federated search can be performed. Compared with the prior art method in which the user first enters a vector index request and then enters a federated search request based on the index result, the complexity of the user query is reduced and the user query experience is improved.

[0035] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without inventive effort.

[0037] Figure 1 A flow chart of a data query method provided by an embodiment of the present disclosure is shown;

[0038] Figure 2 A schematic diagram of a data query device provided by an embodiment of the present disclosure is shown;

[0039] Figure 3 A schematic structural diagram of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0040] In order to make the purposes, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The components of the embodiments of the present disclosure described and shown herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the claimed present disclosure, but only represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present disclosure.

[0041] In addition, the terms "first", "second", and the like in the specification and claims of the embodiments of the present disclosure and the above drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0042] "Multiple or several" mentioned herein refers to two or more. "And / or" describes the association between the associated objects, which means that there can be three relationships, for example, A and / or B, which can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally represents that the associated objects before and after are in an "or" relationship.

[0043] Research has found that federated search, as a technology for executing queries across multiple data sources, breaks down data silos and enables users to perform unified queries across heterogeneous data sources. This not only improves data access efficiency and business decision-making capabilities, but also eliminates the need to integrate data from multiple heterogeneous sources, reducing data replication costs and redundant storage issues while also reducing the complexity of data synchronization and consistency maintenance and simplifying data governance. Furthermore, federated search supports real-time cross-system analysis, increasing data query flexibility. In the era of large-scale neural network models, the similarity search capabilities of vector index queries enable models to quickly retrieve and match multimodal data (text, images, audio, etc.), significantly expanding the model's knowledge base. However, combining vector queries with federated queries often presents the following challenges: 1. Index consistency: Vector index structures may differ across different data sources (for example, some data sources may have native vector indexes while others do not, and even different data sources with vector indexes may have different vector indexes). This makes it difficult to ensure consistent search accuracy when using vector search across different data sources. 2. Standardized interfaces: Vector index query interfaces and similarity calculation algorithms vary across different data sources, making efficient compatibility difficult. 3. Result Fusion: Because vector similarity calculation methods may differ across different data sources, the similarity reported by each data source lacks a unified standard. Consequently, directly sorting and filtering results based on the similarity reported by each data source can affect the accuracy of the results. To address these difficulties, several combined solutions have been proposed. For example, Solution 1 forgoes federated search and instead uses a centralized vector database to store all vector data from different data sources, builds an index within the centralized vector database, and executes queries. Solution 2 combines independent vector indexes with federated queries. Specifically, users execute vector index queries within the vector index. Once the vector query results are obtained, they then enter a federated query request based on the vector query results. However, Solution 1 requires periodic synchronization of vector data from different data sources to the centralized database, increasing storage, computing, and data synchronization costs, and potentially causing data timeliness issues. Furthermore, since federated search is abandoned, efficient federated query support is lacking, making it difficult to efficiently process complex SQL queries or cross-data source data analysis. For solution 2, the user's query process is cumbersome, which increases the query difficulty. In addition, since vector index query is an independent interface and cannot be directly embedded in SQL statements, the processing ability of complex query tasks is limited.

[0044] Based on the above research, the present disclosure provides a data query method and device, computer equipment and program product, by pre-constructing a virtual table for data mapping of a physical table, and setting a vector index column in the virtual table for table data in the target column of the physical table, the vector index query can be realized by locating the target vector index in the virtual table according to the vector index column when querying. Based on the mapping relationship between the target vector index in the virtual table and the physical table, the federated search of the physical table can be realized, thereby improving the efficiency of determining the query result. In this way, based on the virtual table and the vector index column in the virtual table, the efficient combination of federated search and vector index query can be realized. And by introducing a query request with vector query content, the vector query content for vector index query and the SQL content for federated search can be obtained based on the automatic analysis of the user's one-time input query request, and the corresponding vector index query and federated search are performed, compared with the prior art which requires the user to first input the vector index request and then input the federated search request based on the index result, the complexity of user query is reduced, and the user query experience is improved.

[0045] The defects of the above solutions are the results of the inventors after practice and careful study, therefore, the discovery process of the above problems and the solutions proposed by the present disclosure to solve the above problems in the following should be the contributions of the inventors to the present disclosure in the process of the present disclosure.

[0046] It should be noted that: similar reference numbers and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0047] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the type, use range, use scenario, etc. of the personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0048] It should be noted that the specific terms mentioned in the embodiments of the present disclosure include:

[0049] Federated search: a technology for executing queries across multiple data sources, allowing users to access data distributed in different databases, storage systems or services as if querying a single database. Through a unified query interface, data from multiple data sources is dynamically integrated, avoiding data migration, improving query efficiency and system flexibility;

[0050] Vector index query: A method for quickly finding similar data, particularly suitable for complex data such as images, text, or speech. The core idea is to convert data into high-dimensional vector indexes, then use a specific data structure to store and search these vector indexes, making it possible to quickly find the closest matches in large datasets. For example, when a user enters a query request (such as an image or a sentence), the query vector corresponding to the query request can be used to find the most similar results in the vector index.

[0051] MySQL: is a relational database management system that stores data in different tables instead of storing all data in one large warehouse, which increases speed and flexibility.

[0052] Kakfa - a high-throughput distributed publish-subscribe messaging system that handles the streaming of all consumer actions on a website.

[0053] Hive: A data warehouse analysis system built on Hadoop. It provides a rich set of SQL query methods to analyze data stored in the Hadoop distributed file system. It can map structured data files into a database table and provide complete SQL query capabilities.

[0054] To facilitate understanding of this embodiment, a data query method disclosed in an embodiment of the present disclosure is first introduced in detail. The execution subject of the data query method provided in the embodiment of the present disclosure is generally a terminal device or other processing device with a certain computing capability, wherein the terminal device can be a user equipment (UE), a mobile device, a terminal, a personal digital assistant (PDA), a handheld device, a computer device, a virtualization engine deployed in a server, etc.; in some possible implementations, the data query method can be implemented by a processor calling computer-readable instructions stored in a memory.

[0055] The data query method provided by the embodiment of the present disclosure is described below by taking the execution subject as a virtualization engine as an example.

[0056] like Figure 1 FIG. 1 is a flowchart of a data query method provided by an embodiment of the present disclosure, which may include the following steps:

[0057] S101: Parse the query request input by the user to determine the vector query content and structured query language SQL content corresponding to the query request.

[0058] Here, the present application designs an SQL query interface and introduces a special vector query operator (VECTOR_SEARCH) in the standard Structured Query Language (SQL), so that users can directly perform vector searches and federated searches through simple SQL statements. VECTOR_SEARCH), so that users can directly perform vector searches through simple SQL statements. When writing SQL query requests, users do not need to care about the underlying vector index query process. The virtualization engine will automatically parse the vector query operators in the SQL query request and execute the corresponding vector index query tasks. After obtaining the vector query results, a federated search is performed in combination with the SQL content in the query request to obtain the final query results. Specifically, the query process of the present application can include vector index queries and federated searches and SQL operations based on SQL content and vector query results. The entire query process can be completed in a single SQL query request, which greatly simplifies the user's operation process.

[0059] The query request in this application refers to the SQL statement entered by the user by calling the SQL query interface. The query request includes vector query content set based on the vector query operator and SQL query content used for the SQL query. The vector query content can specifically include the query vector used for vector search, and the SQL content can specifically include the SQL statement used to perform SQL operations during federated search.

[0060] In specific implementation, a virtualization engine may be used to receive a user-initiated SQL query request including a vector query operator, and parse the query request to determine the vector query content for vector index search and the SQL content for federated retrieval in the request.

[0061] S102: Perform a vector index query based on the vector query content and the vector index columns in the pre-built virtual table to determine the vector query result; the virtual table is used to map data to at least one physical table and includes a vector index column corresponding to the table data under at least one target column in the physical table; the vector index column is used to store the vector index of the table data under the target column and the table information of the physical table to which the table data belongs.

[0062] A virtual table is an abstraction of a physical table in at least one external data source. Users can use this virtual table to query physical tables in different external data sources. Each virtual table can contain mappings to physical tables in multiple external data sources, and users can selectively create vector indexes for certain columns in the virtual table, simplifying the management of multiple data sources while maintaining the efficiency of vector queries. External data sources can include various structured data sources, such as MySQL databases, Kafka streaming data sources, and Hive databases. A physical table is a data table containing individual rows of data.

[0063] Exemplarily, a virtual table is a source-attached virtual table pre-set for a physical table in at least one external data source. The virtual table does not include specific data in the physical table, but may include data attribute information of the physical table and a vector index column corresponding to the table data under a target column in the physical table. The physical tables in one or more data sources are mapped through the virtual table, so that specific data in the physical table can be located when querying using the virtual table.

[0064] The structure of the virtual table can be determined based on the number of physical tables that need to be mapped. For example, if there is only one physical table that needs to be mapped, a virtual table for mapping the physical table can be constructed based on the data metadata in the physical table (such as data attributes, data fields, data instances, data structure, etc.). If there are multiple physical tables that need to be mapped, a virtual table for mapping each physical table can be constructed based on the data metadata in each physical table. This virtual table can be a large wide table that can store the data mapping relationship between each physical table.

[0065] When constructing a virtual table, at least one target column can be selected to construct a vector index. Using the data vectors of the table data corresponding to the target column in each physical table, the vector indexes corresponding to the target column are generated, resulting in a vector index column. The vector index can include not only the data vectors of the table data under the target column, but also, within the vector index's metadata, table information for the physical table to which the corresponding table data belongs. This table information can specifically include the table identifier of the physical table to which the table data belongs, the column identifier of the column in the physical table to which it belongs, and so on. For example, if the data name column in the virtual table is selected as the target column, then the mapping relationship corresponding to the data name column in the virtual table can be used to index the various table data under the data name column in the physical table. Based on the data vectors of the various table data, the table name of the physical table to which the table data belongs, and the column name of the column to which it belongs, a vector index column corresponding to the data name column in the virtual table is generated. For another example, a user can specify the age column in the virtual table to construct a vector index. The virtualization engine can then convert each specific age under the age field in the physical table mapped by the virtual table into a vector index and store it in the vector index column corresponding to the age in the virtual table. The data vectors and information in the vector index enable the virtual table engine to quickly locate specific columns in the corresponding physical table, thereby efficiently executing vector index queries.

[0066] After the user enters a query request, the virtual table can rewrite the SQL content in the query request into SQL statements for each physical table and perform a federated search on each table. The query results of each physical table are then integrated to obtain the final query result.

[0067] The target column is the data column that the user indicates in the virtual table needs to construct a vector index. The physical table also has a target column containing specific data.

[0068] In specific implementations, a vector index query can be performed based on the vector index content and the vector indexes in the vector index column of the pre-built virtual table to determine the target vector index that matches the vector index content. The target vector index and the table information in the metadata of the target vector index are used as the vector query result.

[0069] In one embodiment, the virtual table can be constructed according to the following steps 1 and 2:

[0070] Step 1: According to a data vector corresponding to table data of at least one pre-specified target column in a plurality of physical tables to be mapped, determine a vector index corresponding to the table data and table information of the physical table corresponding to the vector index.

[0071] Here, the multiple physical tables to be mapped are the physical tables for which a virtual table needs to be constructed, and the target column is a data column in the virtual table for which a vector index needs to be constructed. The data vector corresponding to the table data is obtained by performing vector conversion on the table data. Alternatively, when determining to construct a virtual table, the user can enter the column name for which a vector index needs to be constructed. The virtualization engine can then find the data column corresponding to this column name in the physical table to be mapped as the target column. Furthermore, when constructing the virtual table, the virtual table can also include the target column corresponding to this column name.

[0072] During implementation, users can specify the physical tables to be mapped and the virtual tables to be created for these physical tables. The virtualization engine can then generate data vectors corresponding to the table data in the target columns specified in each physical table to be mapped, based on the table data in the target columns specified in each physical table to be mapped. Based on these data vectors, it can also generate vector indexes corresponding to the target columns. Furthermore, based on the vector index table identifier and the column identifier of the target column, it can determine the table information of the physical table corresponding to the vector index.

[0073] Among them, if a vector index can correspond to one data vector, it can also correspond to multiple data vectors.

[0074] Step 2: Construct a virtual table for mapping multiple physical tables based on data attribute information of multiple physical tables to be mapped and a vector index column composed of vector indexes and table information.

[0075] Here, the data attribute information is used to indicate the attributes of the table data in the physical table, and may include, for example, information such as data fields, field attributes, field names, field examples, and data formats.

[0076] In a specific implementation, the data columns containing the data attribute information of each physical table to be mapped, as well as the mapping relationship of each physical table in the virtual table, can be determined based on the data attribute information of each physical table. Simultaneously, the vector index columns corresponding to the target columns in the virtual table can be generated based on the obtained vector indexes and the table information corresponding to the vector indexes. Based on the data columns, mapping relationships, and index data columns, a virtual table for mapping multiple physical tables is obtained.

[0077] S103: Perform a federated search on the physical table based on the SQL content, the vector query result, and the mapping relationship between the virtual table and the physical table to obtain a data query result.

[0078] In specific implementations, after obtaining the vector query results, the mapping relationship between virtual and physical tables and the table information in the vector query results can be used to locate specific columns in the corresponding target physical table. A subsequent federated search can then be performed based on the SQL content to obtain the data query results. For example, based on the SQL content and the target vector index in the vector query results, federated query statements for each target physical table can be determined. A federated search can then be performed based on these federated query statements to obtain the data query results.

[0079] In this way, by introducing vector query operators and integrating them with traditional SQL queries, users can perform vector index queries in a familiar SQL environment without having to understand the specific implementation of the underlying vector search. This allows non-professional users to easily perform efficient vector index queries, reducing query difficulty.

[0080] In one embodiment, the above S102 may be implemented as follows:

[0081] S102-1: Perform a vector index query based on the vector query content and each vector index in the vector index column to obtain similarities between each vector index and the vector query content.

[0082] In specific implementation, each vector index in the vector index column can be obtained, and based on a similarity calculation algorithm (such as a cosine similarity algorithm), a vector index query can be performed on each vector index and the vector query content to obtain the similarity between each vector index and the vector query content.

[0083] S102-2: Based on similarity, multiple target vector indexes are filtered out from the vector index, and target table information corresponding to each target vector index is obtained from the virtual table; the target table information includes the target physical table where the target table data associated with the target vector index is located and its position in the target physical table.

[0084] Here, the target vector index is the vector index that matches the vector query content, the target table information can be the table information in the metadata of the target vector index, the target physical table can be the physical table indicated by the table identifier in the target table information, the target table data can be the table data mapped by the target vector index, and the target table data is located in the target physical table.

[0085] In a specific implementation, the vector indices can be sorted in descending order based on the similarity between each vector index and the vector query content. A sort order is then determined, and vector indices with a smaller sort order than the set order are used as target vector indices. Alternatively, vector indices with a greater similarity than the set similarity between each vector index and the vector query content can be used as target vector indices. Table information associated with the metadata of each target vector index is retrieved from the virtual table, and this table information is used as the target table information corresponding to the target vector index.

[0086] In one embodiment, a vector index may be associated with data vectors corresponding to multiple table data. When generating a vector index, similar data vectors may be identified based on the similarity between the data vectors of the table data, and a vector index corresponding to these data vectors may be generated based on the commonalities of these data vectors. S102-2 described above may be implemented as follows:

[0087] S102-2-1: Sort the vector indexes in descending order of similarity, and use the vector index with a sorting order smaller than the target order as the target vector index.

[0088] In specific implementation, the similarities between each vector index and the vector query content can be sorted in descending order of similarity to obtain a sorting order, and the vector index with a sorting order less than the target order can be used as the target vector index. Among them, the target order can be set based on experience, or it can be determined based on the total duration of each vector index and the vector query content, or the weight of each vector index can be determined based on the number of data vectors associated with each vector index. The similarities corresponding to each vector index are weighted and summed using each weight, and the ratio of the summation result to the number of vector indexes is determined. The maximum value of this ratio and the number of virtual tables is used as the target order. Alternatively, the target order can also be determined based on the average of the difference between the update time of each vector index and the current time.

[0089] S102-2-2: Based on the similarity between each data vector associated with the target vector index and the vector query content, select the target data vector from the data vectors associated with each target vector index.

[0090] In implementation, for each target vector index, the similarity between each data vector associated with the target vector index and the vector query content can be calculated. Then, the data vector with a similarity greater than a preset similarity is taken as the target data vector. Alternatively, each similarity can be weighted according to the generation time of each data vector to obtain a weighted similarity. Then, the weighted similarities are sorted in descending order, and the data vector with a sorting order less than a target order is taken as the target data vector.

[0091] S102-2-3: Obtain target table information of the target table data corresponding to each target data vector from the virtual table.

[0092] Here, when multiple data vectors are associated with a vector index, the table information stored in the metadata of the vector index can include the table information of each data vector associated therewith.

[0093] In implementation, the target table information corresponding to each target data vector can be obtained from the metadata of each target vector index stored in the virtual table, and each target table information is taken as the target table information corresponding to the corresponding target vector index.

[0094] In this way, by recording the table data metadata in the vector index, the virtualization engine can quickly and accurately locate the relevant data in the cross-data-source query, avoiding the data consistency problem commonly seen in traditional federated queries, not only improving the query accuracy, but also simplifying the management work of cross-data-source, making the multi-source data integration more efficient.

[0095] In an embodiment, the above S103 can be implemented according to the following steps:

[0096] S103-1: Determine each target physical table related to the query request according to each target table information in the vector query result and the mapping relationship.

[0097] Here, the vector query result can include each target vector index and the target table information associated with the target vector index.

[0098] In implementation, each target physical table corresponding to each target vector index can be located according to the target table information associated with each target vector index in the vector query result and the mapping relationship of each target vector index.

[0099] S103-2: Generate a target query statement matching the syntax format of each target physical table according to the SQL content and the target vector index in the vector query result.

[0100] In specific implementations, the SQL content and target vector index in the query request can be converted into target query statements that match the target physical tables based on the SQL content, target vector index, and the SQL syntax format supported by each target physical table. For example, the target vector index can be first converted into index data that matches the SQL syntax format supported by the target physical table. Then, based on the SQL content and index data, a target query statement for performing SQL calculations on the index data can be generated.

[0101] Alternatively, the SQL content and each target data vector may be converted into a target query statement matching each target physical table based on the SQL content in the query request, each target data vector searched under the target vector index, and the SQL syntax format supported by each target physical table.

[0102] S103 - 3 : Using the target query statement corresponding to each target physical table, perform a federated search on each target physical table to obtain a data query result.

[0103] In specific implementation, you can use the target query statement corresponding to each target physical table to perform a federated search on each target physical table to obtain the sub-query results corresponding to each physical table. The results of each sub-query are integrated to obtain the data query result.

[0104] In this way, when a user initiates an SQL query request containing a vector query operator, the virtualization engine first parses the query request, identifies the portion containing the vector search operator, and then performs the vector search, obtaining similarity-ranked results (i.e., the target vector index). The virtualization engine then performs subsequent SQL calculations and federated searches based on these similarity-ranked results and the SQL content, resulting in the data query results. The entire query process automatically handles the combination of vector queries and federated searches targeting structured data, without requiring manual user intervention.

[0105] Furthermore, the data query method provided in this application supports federated queries across data sources. Regardless of the physical table or data source from which the vector index originates, the virtualization engine can accurately locate the table data in the physical table using the index's metadata, avoiding the data consistency issues common in cross-data source queries. This allows users to not only perform vector queries but also integrate and analyze data through SQL queries and federated searches.

[0106] In one embodiment, the present application can also implement automatic updating of vector indexes. Specifically, after constructing a virtual table, in response to inserting new table data into a target column in a physical table, the vector index column in the constructed virtual table can be updated according to the data vector corresponding to the new table data.

[0107] In a specific implementation, whenever new table data is inserted into any target column in any physical table, the virtualization engine can determine the data vector corresponding to the new table data in response to the data insertion operation. Based on this data vector, the vector index in the vector index column of the virtual table is then updated. For example, if a vector index corresponds to a data vector, the vector index corresponding to the new table data can be generated based on the data vector corresponding to the new table data. The table identifier of the physical table where the new table data resides and the column identifier of the target column where the new table data reside are used as the table information for the vector index corresponding to the new table data, and this table information is associated with the metadata of the vector index. The vector index of the new table data and the table information associated with the metadata of the vector index are added to the vector index column of the virtual table. If a vector index corresponds to multiple data vectors, the vector index column can be used to determine whether there is a matching vector index that matches the data vector corresponding to the new table data. If so, the data vector corresponding to the new table data can be associated with the matching vector index, and the table information corresponding to the new table data can be added to the metadata of the matching vector index, thereby updating the vector index column. If not, a vector index corresponding to the new table data can be generated based on the data vector corresponding to the new table data. The table identifier of the physical table where the new table data is located and the column identifier of the target column where the new table data is located are used as the table information of the vector index corresponding to the new table data. The table information is then associated with the metadata of the vector index. The vector index of the new table data and the table information associated with the metadata of the vector index are added to the vector index column of the virtual table.

[0108] In this way, index management can be completed automatically, and users do not need to manually handle the synchronization and query process of each data source, which reduces the operation and maintenance complexity of the virtual table, reduces manual intervention, and reduces the operation and maintenance costs of the product.

[0109] Based on the above embodiments, this application achieves efficient vector search across data sources by embedding vector index queries within a federated search framework. Furthermore, through the metadata management mechanism of vector indexes, queries are accurately mapped to corresponding physical tables.

[0110] In one embodiment, this application embeds vector index query into federated search, making it easier to integrate data query results with artificial intelligence products, thereby improving the efficiency of intelligent applications. Specifically, after obtaining the data query results, you can also:

[0111] Utilize artificial intelligence models to generate query feedback information based on data query results.

[0112] Here, query feedback information is used to provide feedback to objects related to the data query results. For example, if the query results include multiple users, query feedback information can be provided to each user. If the query results include multiple products, query feedback information can be provided to each product. The artificial intelligence model can be any open source model available in the prior art.

[0113] During specific implementation, an artificial intelligence model can be used to determine query feedback information related to the query requirements of the data query request based on the data query request and the target query result.

[0114] For example, if a data query request is for information about a target meeting, and the query results include the attendees and contact information, the AI ​​model can be used to generate meeting notification information for each attendee based on the attendees and contact information, and the meeting notification information can be fed back to each attendee as query feedback. For another example, if a data query request is for order information for a department, and the query results include the order volume and order destinations for each user under the department, the AI ​​model can be used to generate work task schedules for each user based on the order volume and order destinations, and feedback can be provided to each user.

[0115] In this way, this application integrates the existing data from various external data sources through federated queries, which not only effectively solves the problem of data silos, but also provides a low-cost and efficient solution for users to access their existing data into large models.

[0116] Based on the above examples, the present application simplifies cross-data source querying by integrating vector index queries into the federated search framework. Furthermore, through a virtualization engine, vector index queries and structured queries are optimized (for example, users only need to query once to obtain results). This achieves efficient and unified integration of vector index queries and structured queries in a federated search environment, effectively improving the user's data query experience. Because the query and computation processes are automated, the present application can efficiently execute complex query tasks, reducing query latency issues that may occur with traditional methods and significantly improving product response speed and performance. Furthermore, the present application supports queries on both structured and unstructured data (such as vector data), greatly expanding the scope of data analysis. Users can flexibly select which columns to create vector indexes on and write complex SQL queries based on business needs, making the present application applicable to a wider range of application scenarios. Furthermore, the technical solutions of the present application are applicable across multiple industries and fields, such as finance, healthcare, e-commerce, and social networking, particularly those that rely on efficient vector search and complex query analysis, meeting the diverse data analysis and processing needs of different scenarios.

[0117] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0118] Based on the same inventive concept, the embodiment of the present disclosure also provides a data query device corresponding to the data query method. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned data query method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0119] like Figure 2 FIG. 1 is a schematic diagram of a data query device provided by an embodiment of the present disclosure, comprising:

[0120] The parsing module 201 is used to parse the query request input by the user and determine the vector query content and structured query language SQL content corresponding to the query request;

[0121] Determination module 202 is configured to perform a vector index query based on the vector query content and a vector index column in a pre-constructed virtual table to determine a vector query result; the virtual table is configured to perform data mapping on at least one physical table and includes a vector index column corresponding to table data under at least one target column in the physical table; the vector index column is configured to store a vector index of the table data under the target column and table information of the physical table to which the table data belongs;

[0122] The retrieval module 203 is configured to perform a federated search on the physical table according to the SQL content, the vector query result, and the mapping relationship between the virtual table and the physical table to obtain a data query result.

[0123] In a possible implementation, the determining module 202, when performing a vector index query based on the vector query content and the vector index columns in the pre-built virtual table to determine the vector query result, is configured to:

[0124] Performing a vector index query based on the vector query content and each vector index in the vector index column to obtain a similarity between each vector index and the vector query content;

[0125] Based on the similarity, multiple target vector indexes are filtered out from the vector index, and target table information corresponding to each target vector index is obtained from the virtual table; the target table information includes the target physical table where the target table data associated with the target vector index is located and the position in the target physical table.

[0126] In a possible implementation, the vector index is associated with data vectors corresponding to multiple table data;

[0127] The determining module 202, when screening multiple target vector indexes from the vector index based on the similarity and obtaining target table information corresponding to each target vector index from the virtual table, is configured to:

[0128] Sort each vector index in descending order of similarity, and use the vector index with a sorting order smaller than the target order as the target vector index;

[0129] Filtering a target data vector from the data vectors associated with each target vector index based on similarities between each data vector associated with the target vector index and the vector query content;

[0130] The target table information of the target table data corresponding to each target data vector is obtained from the virtual table.

[0131] In a possible implementation, when performing a federated search on the physical table based on the SQL content, the vector query result, and the mapping relationship between the virtual table and the physical table to obtain a data query result, the retrieval module 203 is configured to:

[0132] determining target physical tables related to the query request according to the target table information and the mapping relationships in the vector query result;

[0133] Generate a target query statement matching the syntax format of each target physical table according to the SQL content and the target vector index in the vector query result;

[0134] Using the target query statement corresponding to each target physical table, a federated search is performed on each target physical table to obtain a data query result.

[0135] In a possible implementation, the apparatus further includes a construction module 204 configured to construct the virtual table according to the following steps:

[0136] Determining, based on a data vector corresponding to table data of at least one pre-specified target column in a plurality of physical tables to be mapped, a vector index corresponding to the table data and table information of the physical table corresponding to the vector index;

[0137] A virtual table for mapping the multiple physical tables is constructed according to the data attribute information of the multiple physical tables to be mapped and a vector index column composed of the vector index and the table information.

[0138] In a possible implementation, the apparatus further includes an updating module 205, which, after constructing the virtual table, is configured to:

[0139] In response to inserting new table data into the target column in the physical table, the vector index column in the constructed virtual table is updated according to the data vector corresponding to the new table data.

[0140] In a possible implementation, the apparatus further includes a generating module 206, which, after obtaining the data query result, is configured to:

[0141] Utilize the artificial intelligence model to generate query feedback information based on the data query results.

[0142] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0143] Based on the same technical concept, the embodiment of the present application also provides a computer device. Figure 3 FIG. 1 is a schematic diagram of a computer device according to an embodiment of the present invention, comprising:

[0144] Processor 301, memory 302, and bus 303. The memory 302 stores machine-readable instructions executable by the processor 301, and the processor 301 is configured to execute the machine-readable instructions stored in the memory 302. When the machine-readable instructions are executed by the processor 301, the processor 301 performs the following steps: S101: parsing a query request input by a user to determine vector query content and structured query language (SQL) content corresponding to the query request; S102: performing a vector index query based on the vector query content and vector index columns in a pre-built virtual table to determine a vector query result; the virtual table is configured to perform data mapping on at least one physical table and includes a vector index column corresponding to table data under at least one target column in the physical table; the vector index column is configured to store vector indexes of the table data under the target column and table information of the physical table to which the table data belongs; and S103: performing a federated search on the physical table based on the SQL content, the vector query result, and the mapping relationship between the virtual table and the physical table to obtain a data query result.

[0145] The memory 302 includes an internal memory 3021 and an external memory 3022; the internal memory 3021 is also referred to as an internal storage, used for temporarily storing operation data in the processor 301, and data exchanged with the external memory 3022 such as a hard disk; the processor 301 exchanges data with the external memory 3022 through the internal memory 3021; when the computer device is running, the processor 301 and the memory 302 communicate through the bus 303, so that the processor 301 executes the execution instructions mentioned in the above method embodiments.

[0146] The disclosure embodiments further provide a computer readable storage medium, which stores a computer program, and the computer program is run by a processor to perform the steps of the data query method described in the above method embodiments. The storage medium can be a volatile or non-volatile computer readable storage medium.

[0147] The disclosure embodiments further provide a computer program product, which carries a program code, and the program code includes instructions for performing the steps of the software update method described in the above method embodiments, which can be referred to the above method embodiments and will not be repeated here.

[0148] The computer program product can be specifically implemented by hardware, software or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium, and in another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.

[0149] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system and device can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here. In several embodiments provided by the disclosure, it should be understood that the disclosed system, device and method can be implemented by other ways. The above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and other division manners can be used in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interfaces, devices or units, and can be electrical, mechanical or other forms.

[0150] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0151] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0152] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0153] If the technical solution of this application involves personal information, the product that applies the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product that applies the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0154] Finally, it should be noted that the above-described embodiments are merely specific embodiments of the present disclosure, used to illustrate the technical solutions of the present disclosure, and are not intended to limit the present disclosure. The protection scope of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easy changes to the technical solutions described in the foregoing embodiments, or easily think of changes or equivalent replacements for some of the technical features; and these modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A data query method, characterized in that: include: Parsing the query request input by the user to determine the vector query content and structured query language SQL content corresponding to the query request; Performing a vector index query based on the vector query content and a vector index column in a pre-built virtual table to determine a vector query result; the virtual table is used to perform data mapping on at least one physical table and includes a vector index column corresponding to table data under at least one target column in the physical table; The vector index column is used to store the vector index of the table data under the target column and the table information of the physical table to which the table data belongs; Performing a federated search on the physical table according to the SQL content, the vector query result, and the mapping relationship between the virtual table and the physical table to obtain a data query result; The performing a vector index query based on the vector query content and the vector index columns in the pre-built virtual table to determine the vector query result includes: Performing a vector index query based on the vector query content and each vector index in the vector index column to obtain a similarity between each vector index and the vector query content; Based on the similarity, multiple target vector indexes are filtered out from the vector index, and target table information corresponding to each target vector index is obtained from the virtual table; the target table information includes the target physical table where the target table data associated with the target vector index is located and the position in the target physical table.

2. The method according to claim 1, characterized in that The vector index is associated with a plurality of data vectors corresponding to table data; The step of screening out a plurality of target vector indexes from the vector index based on the similarity, and acquiring target table information corresponding to each target vector index from the virtual table includes: Sort each vector index in descending order of similarity, and use the vector index with a sorting order smaller than the target order as the target vector index; Filtering a target data vector from the data vectors associated with each target vector index based on similarities between each data vector associated with the target vector index and the vector query content; The target table information of the target table data corresponding to each target data vector is obtained from the virtual table.

3. The method according to claim 1, characterized in that The step of performing a federated search on the physical table according to the SQL content, the vector query result, and the mapping relationship between the virtual table and the physical table to obtain a data query result includes: determining target physical tables related to the query request according to the target table information and the mapping relationships in the vector query result; Generate a target query statement matching the syntax format of each target physical table according to the SQL content and the target vector index in the vector query result; Using the target query statement corresponding to each target physical table, a federated search is performed on each target physical table to obtain a data query result.

4. The method according to claim 1, wherein The virtual table is constructed according to the following steps: Determining, based on a data vector corresponding to table data of at least one pre-specified target column in a plurality of physical tables to be mapped, a vector index corresponding to the table data and table information of the physical table corresponding to the vector index; A virtual table for mapping the multiple physical tables is constructed according to the data attribute information of the multiple physical tables to be mapped and a vector index column composed of the vector index and the table information.

5. The method according to claim 4, characterized in that After constructing the virtual table, the following steps are also included: In response to inserting new table data into the target column in the physical table, the vector index column in the constructed virtual table is updated according to the data vector corresponding to the new table data.

6. The method according to claim 1, characterized in that After obtaining the data query results, it also includes: Utilize the artificial intelligence model to generate query feedback information based on the data query results.

7. A data query device, characterized in that: include: A parsing module, configured to parse a query request input by a user and determine the vector query content and structured query language SQL content corresponding to the query request; a determination module, configured to perform a vector index query based on the vector query content and vector index columns in a pre-built virtual table, and determine a vector query result; the virtual table is configured to perform data mapping on at least one physical table and includes a vector index column corresponding to table data under at least one target column in the physical table; The vector index column is used to store the vector index of the table data under the target column and the table information of the physical table to which the table data belongs; the determination module, when performing a vector index query based on the vector query content and the vector index column in the pre-built virtual table to determine the vector query result, is used to: Performing a vector index query based on the vector query content and each vector index in the vector index column to obtain a similarity between each vector index and the vector query content; Based on the similarity, a plurality of target vector indexes are screened from the vector indexes, and target table information corresponding to each target vector index is obtained from the virtual table; the target table information includes a target physical table where the target table data associated with the target vector index is located and a position in the target physical table; A retrieval module is used to perform a federated search on the physical table according to the SQL content, the vector query result, and the mapping relationship between the virtual table and the physical table to obtain a data query result.

8. A computer device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor performs the steps of the data query method according to any one of claims 1 to 6.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a computer device, the computer device executes the steps of the data query method according to any one of claims 1 to 6.

Images