Data processing method, server, storage medium and program product

Abstract

The invention provides a data processing method, a server, a storage medium and a program product. The method relates to a computer technology, and comprises the following steps: in a multi-database and mass data table scene, in a global range of databases, according to description information of each database and description information of a table connection sub-graph of each database, roughly recalling related candidate databases; according to the natural language query, the description information of each data table in the candidate database and the description information of each table column, finely sorting the candidate database, determining a target database, a target data table and a target table column associated with the natural language query according to a fine sorting result, and combining rough recall and multi-level fine sorting to obtain a target database; according to the method, the user query is intelligently routed to the associated target database, target data table and target table column, the global data query requirement of the user is met, complex data treatment and scene isolation do not need to be depended on, the multi-database query process is simplified through uniform intelligent routing, and the operation convenience of multi-database query is improved.

Description

Technical Field

[0001] This application relates to computer technology, and particularly to a data processing method, a server, a storage medium, and a program product. Background Art

[0002] With the rapid development of information technology, data-driven decision-making plays an increasingly important role in all walks of life. As the core tool for storing and managing a large amount of structured data, the efficient access and operation capabilities of databases are crucial for optimizing system processes. However, traditional database queries rely on Structured Query Language (SQL for short), which requires users to have certain programming knowledge and skills, restricting the direct data access and analysis capabilities of non-technical personnel. To address this bottleneck, Natural Language Processing (NLP for short) technology has gradually been applied to the field of database queries, giving rise to the research and application of Natural Language to SQL (NL2SQL for short).

[0003] NL2SQL refers to the conversion of natural language queries into SQL queries that can be executed on relational databases, which is a typical application that combines natural language processing and database management. The core goal of NL2SQL is to understand the query intent expressed by users in natural language and generate SQL statements that accurately reflect this intent, enabling users to directly interact with databases efficiently through natural language without having to master complex SQL syntax. This technology not only lowers the threshold for database access but also broadens the user group for data analysis, with broad application prospects.

[0004] In recent years, the emergence of Large Language Models (LLMs for short) has injected new vitality into the development of NL2SQL technology. Compared with traditional methods, LLMs excel in understanding complex language structures, semantic reasoning, and generation capabilities, significantly improving the performance of NL2SQL systems. These large models can capture rich language patterns and knowledge through large-scale data training, thereby more accurately understanding user intent and generating SQL queries that conform to the database schema. In addition, the use of LLMs enables NL2SQL systems to support spoken dialogues, further simplifying the interaction process between users and databases and providing new possibilities for commercial applications and system process reforms.

[0005] Products of the Chat-based Business Intelligence (ChatBI) category are generated based on the NL2SQL technology, enabling users to easily perform data query and analysis in a conversational manner. Currently, ChatBI products on the market are oriented towards application-level databases (i.e., one application corresponds to one database / data source, and the number of data tables in a database is less than 100, i.e., DataBases = 1, Tables < 100), and perform queries in a single data source based on user queries. When facing applications with multiple databases, they rely on data governance and data isolation and cannot support users' query requirements for the entire database. Summary of the Invention

[0006] This application provides a data processing method, a server, a storage medium, and a program product to solve the problem that the query system based on NL2SQL cannot support users' query requirements for the entire database when facing applications with multiple databases.

[0007] In a first aspect, this application provides a data processing method, including:

[0008] Receiving an input natural language query; recalling candidate databases related to the natural language query according to the description information of each database and the description information of the table connectivity subgraphs of each of the databases; performing a refined sorting on the candidate databases according to the natural language query, the description information of each data table in the candidate databases, and the description information of each table column, to obtain a refined sorting result; selecting at least one of the candidate databases as the target database associated with the natural language query according to the refined sorting result; and determining a target data table and target table columns associated with the natural language query in the target database.

[0009] In a second aspect, this application provides a data processing method, including:

[0010] Responding to a natural language to SQL conversion request, obtaining the natural language query to be converted; recalling candidate databases related to the natural language query according to the description information of each database in the enterprise-level database and the description information of the table connectivity subgraphs of each of the databases; performing a refined sorting on the candidate databases according to the natural language query, the description information of each data table in the candidate databases, and the description information of each table column, to obtain a refined sorting result; selecting at least one of the candidate databases as the target database associated with the natural language query according to the refined sorting result; determining a target data table and target table columns associated with the natural language query in the target database; generating an SQL statement corresponding to the natural language query according to the target database, the target data table, and the target table columns; and outputting the SQL statement.

[0011] In a third aspect, the present application provides a server, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the server is caused to execute the method provided in any of the foregoing aspects.

[0012] In a fourth aspect, the present application provides a computer-readable storage medium storing computer-executable instructions, and when a processor executes the computer-executable instructions, the method provided in any of the foregoing aspects is implemented.

[0013] In a fifth aspect, the present application provides a computer program product including a computer program, and when the computer program is executed by a processor, the method provided in any of the foregoing aspects is implemented.

[0014] The data processing method, server, storage medium and program product provided by the present application recall candidate databases related to the natural language query according to the description information of each database and the description information of the table connectivity subgraphs of each of the databases; perform fine sorting on the candidate databases according to the natural language query, the description information of each data table in the candidate databases and the description information of each table column to obtain a fine sorting result; select at least one of the candidate databases according to the fine sorting result as the target database associated with the natural language query; and determine the target data table and target table column associated with the natural language query in the target database. In a scenario of multiple databases and a vast number of data tables, based on an intelligent routing strategy, within the entire domain of the databases, based on the unified description information of the databases, the description information of the table connectivity subgraphs of the databases, the description information of the data tables and the description information of the table columns, the user query is intelligently routed to the target database, target data table and target table column associated with the user query within the entire domain, realizing the user's demand for querying data across the entire domain, and without relying on complex data governance and scenario isolation. Through unified intelligent routing, the process of multi-database query is simplified, and the operation convenience of multi-database query is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0016] Figure 1 A schematic diagram of ChatBI query based on data governance and data isolation;

[0017] Figure 2 A schematic diagram of realizing querying data across the entire domain provided by an embodiment of the present application;

[0018] Figure 3 Flowchart of a data processing method provided by an exemplary embodiment of the present application;

[0019] Figure 4 Flowchart of a rough recall candidate database provided by an exemplary embodiment of the present application;

[0020] Figure 5 Flowchart of fine ranking provided by an exemplary embodiment of the present application;

[0021] Figure 6 An example framework diagram of intelligent routing provided by an embodiment of the present application;

[0022] Figure 7 Flowchart of a data processing method provided by another exemplary embodiment of the present application;

[0023] Figure 8 Schematic structural diagram of a server provided by an embodiment of the present application.

[0024] Through the above-mentioned drawings, specific embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. Detailed implementation manners

[0025] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0026] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards, and corresponding operation entrances are provided for users to choose to authorize or reject.

[0027] First, the nouns involved in the present application are explained:

[0028] Application-level database: The data base generally relied on by NL2SQL systems based on LLM. A single database contains a hundred-level number of data tables.

[0029] Enterprise-level database: An industrial database for enterprise-level real applications, including a large number of databases, and a large number of data tables in a single database. The number of databases in an enterprise-level database is usually greater than 50, and the number of tables is usually greater than 5000.

[0030] Intelligent routing: Automatically recall relevant databases, tables, and fields (i.e., table columns) from the enterprise-level database according to the natural language query input by the user.

[0031] In ChatBI queries for enterprise-level databases, the scale of enterprise databases is often very large. Usually, the number of databases (denoted as Databases) is greater than 10, and the number of data tables in each database (denoted as Tables) is greater than 500. The user's query (i.e., Query) can match the corresponding query requirements in multiple tables of multiple databases.

[0032] Currently, ChatBI products on the market are for application-level databases, that is, one application corresponds to one database / data source, and the number of data tables in a database is less than 100, that is, DataBases = 1, Tables < 100. Figure 1 It is a schematic diagram of ChatBI query based on data governance / data isolation. As Figure 1 shown, one application corresponds to one database, and the number of data tables in each database (denoted as Tables) does not exceed 100. For example, Figure 1 the application 1DB, application 2DB,... shown in it represent the databases corresponding to different applications 1, 2,... respectively, where the number of data tables in each database (denoted as Tables) is less than 100. Currently, all ChatBI products are for application-level databases. Based on the user's Query, NL2SQL processing can only be performed in the data source of a single application to generate SQL and execute the SQL query. The query results of different application-level databases are summarized, and a summary is generated for the summary results to obtain the response results of the user's query.

[0033] When applying to enterprise-level databases, it relies on complex data governance and data isolation, and the enterprise-level database needs to be split into multiple application-level databases, which cannot support the user's query requirements for the entire database.

[0034] To solve the above technical problems, the present application provides a data processing method. According to the description information of each database in the enterprise-level database and the description information of the table connection subgraph of each database, candidate databases related to the natural language query are recalled; according to the natural language query, the description information of each data table in the candidate databases, and the description information of each table column, the candidate databases are finely sorted to obtain a fine sorting result; at least one candidate database is selected according to the fine sorting result as the target database associated with the natural language query; the target data table and target table column associated with the natural language query are determined in the target database.

[0035] The solution of the embodiment of the present application, in the face of multiple databases and a large number of data tables in the enterprise-level database, based on an intelligent routing (Schema Routing) strategy, within the entire range of the enterprise-level database, based on the unified description information of the database, the description information of the table connection subgraph of the database, the description information of the data table, and the description information of the table column, intelligently routes the user query to the target database, target data table, and target table column associated with the user query within the entire range, realizes the user's demand for global data query, and does not require relying on complex data governance and scenario isolation. Through unified intelligent routing, the process of multi-database query is simplified, and the operation convenience of multi-database query is improved.

[0036] Figure 2 It is a schematic diagram for realizing global data query provided by the embodiment of the present application. As Figure 2 shown, based on the input user query, based on the above intelligent routing (Schema Routing) strategy, within the entire range of the enterprise-level database, intelligent routing is performed based on the unified description information of the database, the description information of the table connection subgraph of the database, the description information of the data table, and the description information of the table column to determine the target database, target data table, and target table column associated with the user query, realizes the user's demand for global data query, and does not require relying on complex data governance and scenario isolation, simplifies the process of multi-database query, and improves the operation convenience of multi-database query. Further, NL2SQL processing is performed based on the target database, target data table, and target table column to generate SQL. Further, by executing the SQL query, summarizing the query results, and generating a summary, the response result of the user query can be obtained.

[0037] The technical solution of the present application and how the technical solution of the present application solves the above technical queries will be described in detail below with specific embodiments. These specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0038] Figure 3Flowchart of a data processing method provided by an exemplary embodiment of the present application. The execution subject of this embodiment may be a server for implementing data query, such as a server of NL2SQL, or a data query server based on NL2SQL, etc. There is no specific limitation here in this embodiment. As Figure 3 shown, the specific steps of this method are as follows:

[0039] Step S301: Receive the input natural language query.

[0040] In the ChatBI query scenario based on the enterprise-level database, the user inputs a natural language query (i.e., user Query) through the end-side device. The end-side device sends the input natural language query to the server, requesting the server to convert the natural language query into an SQL statement that can be executed in the database.

[0041] In an example scenario, the server can return the SQL statement corresponding to the natural language query to the end-side device. The end-side device can execute the SQL statement in the local database to obtain the corresponding query result. The end-side device can output the query result to the user. The end-side device can also generate a natural language reply message based on the query result and return the reply message to the user.

[0042] In another example scenario, the database can be deployed on the server side. After the server converts the user's natural language query into an SQL statement, it executes the SQL statement in the database to obtain the corresponding query result. The server can return the output query result to the end-side device. The server can also generate a natural language reply message based on the query result and return the reply message to the end-side device. The end-side device can output the query result or reply message returned by the server to display the query result or reply message to the user.

[0043] In this embodiment, the natural language query can be the original natural language text input by the user, or the content text converted from the user's voice question. There is no specific limitation on the input method of the natural language query here. In the ChatBI query scenario based on the enterprise-level database, the scale of the databases and data tables in the enterprise database is relatively large. Usually, the user's query Query can match multiple tables in multiple databases. The solution of this embodiment is applicable to various data query scenarios across multiple databases or NL2SQL scenarios across multiple databases.

[0044] Step S302: Recall candidate databases related to the natural language query according to the description information of each database and the description information of the table connection subgraph of each database.

[0045] In this step, the server performs a rough recall of the database based on the natural language query input by the user. Among multiple databases to be queried (such as the databases included in the enterprise-level database), it quickly recalls the databases related to the natural language query. In this embodiment, the data related to the natural language query roughly recalled in this step is referred to as candidate databases.

[0046] Among them, the description information of the database includes: the description text of the database and / or the description vector (i.e., the high-dimensional vector representation of the description text). The description information of the table connected subgraph of the database includes: the description text of the table connected subgraph and / or the description vector of the table connected subgraph.

[0047] In an alternative embodiment, the server may adopt a multi-path recall strategy to achieve a rough recall of candidate databases based on the natural language query.

[0048] Exemplarily, the server may adopt a text similarity recall strategy and a vector similarity recall strategy to perform a two-way rough recall, obtain two-way recall results, and use the databases in both two-way recall results as candidate databases.

[0049] Among them, when the server adopts the text similarity recall strategy to perform a rough recall of candidate databases, it calculates the text similarity between the natural language query and each database according to the natural language query, the description text of each database, and the description text of the table connected subgraph of each database. It sorts each database according to the text similarity with the natural language query, and filters out the databases with a relatively large text similarity with the natural language query according to the sorting result as the recalled candidate databases. For example, it recalls the first quantity (such as TOP15) of databases with a relatively large text similarity with the natural language query. The first quantity can be set and adjusted according to the needs of the actual application scenario, and no specific limitation is made here.

[0050] Optionally, when calculating the text similarity between the natural language query and each database, the server may calculate the first text similarity between the natural language query and the description text of each database respectively, and the second text similarity between the natural language query and the description text of the table connected subgraph of each database respectively; and comprehensively calculate the text similarity between the natural language query and each database based on the first text similarity and the second text similarity.

[0051] Optionally, when calculating the text similarity between the natural language query and each database, the server may use the first text similarity between the natural language query and the description text of each database respectively as the text similarity between the natural language query and each database. Optionally, when calculating the text similarity between the natural language query and each database, the server may use the second text similarity between the natural language query and the description text of the table connected subgraph of each database respectively as the text similarity between the natural language query and each database.

[0052] When the server performs rough recall of the candidate database using the vector similarity recall strategy, it calculates the vector similarity between the natural language query and each database based on the vector representation of the natural language query, the description vectors of each database, and the description vectors of the table connectivity subgraphs of each database. Further, it sorts each database according to the vector similarity with the natural language query and filters out the databases with a relatively large vector similarity with the natural language query as the candidate databases for recall. For example, it recalls the second number (such as TOP15) of databases with a relatively large vector similarity with the natural language query. Here, the second number and the first number can be equal or not equal, and can be specifically set according to actual application requirements, and no specific limitation is made here.

[0053] Optionally, when calculating the vector similarity between the natural language query and each database, the server can convert the natural language query into a vector representation, calculate the first vector similarity between the vector representation of the natural language query and the description vectors of each database respectively, and the second vector similarity between the vector representation of the natural language query and the description vectors of the table connectivity subgraphs of each database respectively; and comprehensively calculate the vector similarity between the natural language query and each database based on the first vector similarity and the second vector similarity.

[0054] Optionally, when calculating the vector similarity between the natural language query and each database, the server can use the first vector similarity between the vector representation of the natural language query and the description vectors of each database as the vector similarity between the natural language query and each database. Optionally, when calculating the vector similarity between the natural language query and each database, the server can use the second vector similarity between the vector representation of the natural language query and the description vectors of the table connectivity subgraphs of each database as the vector similarity between the natural language query and each database.

[0055] In another alternative implementation, the server can adopt a recall strategy that includes a text similarity recall strategy and a vector similarity recall strategy, as well as at least one other recall strategy, such as ES (Elasticsearch) recall, etc., and comprehensively use a recall strategy with more paths to recall the candidate databases related to the natural language query, which can recall the candidate databases related to the natural language query more comprehensively.

[0056] In another alternative implementation, the server can adopt a single-path recall strategy to perform rough recall of the candidate database based on the natural language query. Exemplarily, the server can adopt any one of the foregoing text similarity recall strategy and vector similarity recall strategy, or adopt other recall strategies (such as ES recall) to implement rough recall of the candidate database, and no specific limitation is made here in this embodiment.

[0057] It should be noted that when calculating the text similarity between a natural language query and any description text, any method for calculating the similarity between two texts can be used. For example, text similarity calculation methods based on Edit Distance, the longest common subsequence, Term Frequency-Inverse Document Frequency (TF-IDF), etc. are not specifically limited in this embodiment.

[0058] It should be noted that when calculating the vector similarity between the vector representation of a natural language query and any description vector, any method for calculating the similarity between two vector representations can be used. For example, cosine similarity, Euclidean distance, Jaccard Similarity, etc. are not specifically limited in this embodiment.

[0059] In this embodiment, in the rough recall stage, by combining multi-path recall strategies such as vector matching, text matching, and ES similarity, and extending to vector matching and text matching based on connected subgraphs, relevant candidate databases can be quickly screened out, covering a wider range of data sources and query requirements, enhancing query coverage, ensuring high-coverage query results, and meeting the diverse needs of enterprise users.

[0060] ‌Step S303: According to the natural language query, the description information of each data table in the candidate database, and the description information of each table column, perform fine sorting on the candidate database to obtain a fine sorting result.

[0061] Among them, the description information of the data table includes: the description text of the data table and / or the description vector of the data table. The description information of the table column includes: the description text of the table column and the description vector of the table column.

[0062] For the candidate databases related to the natural language query in the rough recall result, the server uses a hierarchical fine sorting logic based on the natural language query, the description information of each data table in the candidate database, and the description information of each table column to evaluate the relevance between the database and the natural language query at multiple levels such as data tables and table columns in the database, calculate the fine sorting score of the candidate database and the natural language query, and perform fine sorting on the candidate database according to the fine sorting score to obtain the fine sorting result of the candidate database.

[0063] In an alternative embodiment, when implementing the fine sorting of the candidate database, the server may calculate the third text similarity between the natural language query and the description text of each data table in the candidate database, and the fourth text similarity between the natural language query and the description text of each table column in the candidate database; according to the third text similarity and the fourth text similarity, calculate the text similarity score of the candidate database as a fine sorting score. Sort the candidate database according to the text similarity score of the candidate database to obtain the fine sorting result of the candidate database.

[0064] Optionally, when calculating the text similarity score of the candidate database according to the third text similarity and the fourth text similarity, the mean (or median) of the third text similarities corresponding to each data table in the candidate database and the mean (or median) of the fourth text similarities corresponding to each table column in the candidate database may be calculated, and the mean (or median) of the third text similarities is added to the mean (or median) of the fourth text similarities to obtain the text similarity score of the candidate database.

[0065] Optionally, when calculating the text similarity score of the candidate database according to the third text similarity and the fourth text similarity, the third text similarities corresponding to each data table in the candidate database are filtered according to the third threshold to determine the third text similarities greater than or equal to the third threshold; the fourth text similarities corresponding to each table column in the candidate database are filtered according to the fourth threshold to determine the fourth text similarities greater than or equal to the fourth threshold; calculate the sum of the mean (or median) of the third text similarities greater than or equal to the third threshold and the mean (or median) of the fourth text similarities greater than or equal to the fourth threshold to obtain the text similarity score of the candidate database. The third threshold and the fourth threshold may be configured according to the requirements of the actual application scenario and are not specifically limited here.

[0066] In another alternative embodiment, when implementing the fine sorting of the candidate database, the server may calculate the third vector similarity between the vector representation of the natural language query and the description vector of each data table in the candidate database, and the fourth vector similarity between the vector representation of the natural language query and the description vector of each table column in the candidate database; according to the third vector similarity and the fourth vector similarity, calculate the vector similarity score of the candidate database as a fine sorting score. Sort the candidate database according to the vector similarity score of the candidate database to obtain the fine sorting result of the candidate database.

[0067] Among them, calculating the vector similarity score of the candidate database according to the third vector similarity and the fourth vector similarity is similar to the implementation principle of calculating the text similarity score of the candidate database according to the third text similarity and the fourth text similarity in the previous alternative embodiment, and will not be elaborated here.

[0068] In another alternative embodiment, when implementing the fine sorting of the candidate database, the server may calculate at least one fine sorting score of the candidate database: Reciprocal Rank Fusion (RRF) score, table vector similarity score, table column vector similarity score, and table priority score. Determine the comprehensive fine sorting score of the candidate database according to at least one fine sorting score of the candidate database. Sort the candidate databases according to the comprehensive fine sorting score of the candidate database to obtain the fine sorting result.

[0069] In this embodiment, based on the candidate database obtained by rough recall, the RRF method is used to comprehensively calculate the RRF score of the candidate database in different recall strategies, and combined with the table priority, table vector similarity, and table column vector similarity, the candidate database obtained by rough recall is accurately sorted to achieve hierarchical fine sorting; and the calculation of the fine sorting scores of different candidate databases can be executed in parallel, which can improve the efficiency of data query and the efficiency of NL2SQL. The specific implementation principle of this embodiment will be explained in detail in the subsequent embodiments.

[0070] It should be noted that the calculation process of the fine sorting score of the database can be executed in parallel with the process of rough recall of the database. When performing rough recall of the database, the data relied on is the natural language query, the description information of each database, and the description information of the table connectivity subgraph of each database. In the calculation process of the fine sorting score of the database, except that the RRF score of the database needs to rely on the sorting result of rough recall, other fine sorting scores (including text similarity scores, various vector similarity scores, table priority scores, etc.) do not need to rely on the result of rough recall. Therefore, a parallel execution mechanism of rough recall and hierarchical fine sorting can be adopted, which can greatly improve the efficiency of data query and the efficiency of NL2SQL, achieve a millisecond-level query response in a multi-database environment, can efficiently process large-scale data requests, greatly shorten the user waiting time, and improve the real-time performance and interactivity of the system.

[0071] The solution of this embodiment adopts a strategy of combining multi-path rough recall and hierarchical fine sorting, which significantly improves the accuracy and coverage rate of recalling relevant table column information in a large number of databases and data tables, thereby improving the accuracy of data query.

[0072] Step S304: Select at least one candidate database according to the fine sorting result as the target database associated with the natural language query.

[0073] In this step, the server determines at least one candidate database with a higher ranking according to the fine sorting result of the candidate database as the target database associated with the natural language query.

[0074] Exemplarily, the server may, according to the refined sorting result of the candidate databases, determine the candidate databases with the top third quantity (such as TOP5) in the sorting as the target databases associated with the natural language query. Herein, the third quantity may be set according to the requirements of the actual application scenario. For example, the third quantity may be set to 3, 5, etc., and no specific limitation is made herein in this embodiment.

[0075] Step S305: Determine the target data table and the target table column associated with the natural language query in the target database.

[0076] After determining the target database associated with the natural language query, the target data table and the target table column associated with the natural language query may be further screened and determined in the target database.

[0077] In an optional implementation manner, the server may determine the target data table in the target database according to the matching degree between the natural language query and the description information of each data table in the target database. Further, the server may determine the target table column in the target data table according to the matching degree between the natural language query and the description information of each table column in the target data table.

[0078] Specifically, for any target database, when determining the target data table in the target database, according to the natural language query and the description information of each data table in the target database, calculate the matching degree between the description information of each data table and the natural language query, which is simply referred to as the matching degree corresponding to the data table. Determine the data table with a relatively high corresponding matching degree as the target data table. For example, sort the data tables in the target database according to the matching degree corresponding to the data tables, and select the data tables with the top fourth quantity (such as TOP5) in the sorting as the target data tables; or, according to the matching degree corresponding to the data tables, select the data tables with a matching degree greater than the fifth threshold as the target data tables. Herein, both the fourth quantity and the fifth threshold may be set according to the needs and experience of the actual application scenario, and no specific limitation is made herein.

[0079] Optionally, for any target database, when calculating the matching degree corresponding to each data table in the target database, the server may use the third text similarity between the natural language query and the description text of each data table in the target database as the matching degree corresponding to each data table.

[0080] Optionally, for any target database, when calculating the matching degree corresponding to each data table in the target database, the server may use the third vector similarity between the vector representation of the natural language query and the description vector of each data table in the target database as the matching degree corresponding to each data table.

[0081] Optionally, for any target database, when calculating the matching degree corresponding to each data table in the target database, the server may perform a weighted summation of the third text similarity and the third vector similarity corresponding to the data table to obtain the matching degree corresponding to the data table. The weight coefficients of the third text similarity and the third vector similarity may be set and adjusted according to the requirements of the actual application scenario, and are not specifically limited in this embodiment.

[0082] For any target data table, when determining the target table columns in the target data table, the matching degree between the description information of each table column and the natural language query is calculated based on the natural language query and the description information of each table column in the target data table, which is referred to as the matching degree corresponding to the table column. The table column with a higher corresponding matching degree is determined as the target table column. For example, the table columns in the target data table are sorted according to the matching degree corresponding to the table columns, and the top fifth number of table columns (such as TOP10) are selected as the target table columns; or, according to the matching degree corresponding to the table columns, a data table with a matching degree greater than a sixth threshold is selected as the target data table. The fifth number and the sixth threshold can be set according to the needs and experience of the actual application scenario, and are not specifically limited here.

[0083] Optionally, for any target data table, when calculating the matching degree corresponding to each table column in the target data table, the server may use the fourth text similarity between the natural language query and the description text of each table column in the target data table as the matching degree corresponding to each table column.

[0084] Optionally, for any target data table, when calculating the matching degree corresponding to each table column in the target data table, the server may use the fourth vector similarity between the vector representation of the natural language query and the description vector of each table column in the target data table as the matching degree corresponding to each table column.

[0085] Optionally, for any target data table, when calculating the matching degree corresponding to each table column in the target data table, the server may weightedly sum the fourth text similarity and the fourth vector similarity corresponding to the table column to obtain the matching degree corresponding to the table column. The weight coefficients of the fourth text similarity and the fourth vector similarity may be set and adjusted according to the requirements of the actual application scenario, and are not specifically limited in this embodiment.

[0086] In another optional implementation, the server may determine the target table column in the target database according to the matching degree between the natural language query and the description information of each table column in the target database, and use the data table where the target table column is located as the target data table.

[0087] In addition, after the server quickly locates the target database, target data table, and target table column associated with the natural language query based on the input natural language query through a unified intelligent routing policy, it can also execute subsequent processing logic according to the processing logic configured in the system based on the target database, target data table, and target table column to implement functions such as NL2SQL and ChatBI queries. Specifically, it can be developed and designed according to the requirements of the actual application scenario, and no specific limitation is made here.

[0088] In an example scenario, after the server quickly locates the target database, target data table, and target table column associated with the natural language query based on the input natural language query through a unified intelligent routing policy, it can also generate a structured query language (SQL) statement corresponding to the natural language query according to the target database, target data table, and target table column, and output the SQL statement to implement the NL2SQL function.

[0089] In another example scenario, after the server quickly locates the target database, target data table, and target table column associated with the natural language query based on the input natural language query through a unified intelligent routing policy, it can also generate a structured query language (SQL) statement corresponding to the natural language query according to the target database, target data table, and target table column, execute the SQL statement to obtain the query result, and output the query result to implement the ChatBI query function across multiple databases.

[0090] In another example scenario, after the server quickly locates the target database, target data table, and target table column associated with the natural language query based on the input natural language query through a unified intelligent routing policy, it can also generate a structured query language (SQL) statement corresponding to the natural language query according to the target database, target data table, and target table column, execute the SQL statement to obtain the query result, generate a reply message according to the query result, and output the reply message to implement the ChatBI query function across multiple databases.

[0091] It should be noted that in a multi-database scenario, the NL2SQL conversion tasks and data query tasks of multiple target databases can be performed in parallel, and the corresponding query tasks can be quickly located and scheduled among multiple databases to achieve parallel processing, significantly improving the query response speed and shortening the overall query response time.

[0092] The solution of this embodiment, in the scenario of multiple databases and a vast number of data tables, based on a unified intelligent routing (Schema Routing) strategy, and based on the description information of the unified database, the description information of the table connectivity subgraph of the database, the description information of the data table, and the description information of the table columns, intelligently routes the user query to the target database, target data table, and target table columns associated with the user query within the entire database scope, realizes the user's demand for querying the entire domain data (across multiple databases), and without relying on complex data governance and scenario isolation, simplifies the process of multi-database query through unified intelligent routing, and improves the accuracy and operation convenience of multi-database query.

[0093] In an optional embodiment, the server can utilize a description generation model and graph theory to automatically generate database description information, table description information, table connectivity subgraph description information, and corresponding high-dimensional vector representations based on the column information of the data tables in the database and their topological relationships, to support subsequent similarity calculation, rough recall, and multi-level fine ranking. This process realizes full automation without manual intervention, significantly improving the data processing efficiency and accuracy.

[0094] In this embodiment, the server automatically generates the description information of each database, the description information of the table connectivity subgraph of each database, the description information of the data tables in each database, and the description information of the table columns in each database. Among them, the description information of the database includes: the description text and / or description vector of the database. The description information of the table connectivity subgraph of the database includes: the description text and / or description vector of the table connectivity subgraph. The description information of the data table includes: the description text and / or description vector of the data table. The description information of the table column includes: the description text and description vector of the table column.

[0095] Optionally, the server can also update the description information of each database, the description information of the table connectivity subgraph of each database, the description information of the data tables in each database, and the description information of the table columns in each database at regular intervals. Specifically, as the historical queries increase and the table and column information of the database is updated, the server can regularly regenerate the description information of each database, the description information of the table connectivity subgraph of each database, the description information of the data tables in each database, and the description information of the table columns in each database based on the historical queries (including incremental queries) of each database and data table, and the latest table and column information of each database, realizing the full automation update of these description information without manual intervention, significantly improving the data processing efficiency and accuracy. By continuously automatically updating the description information based on incremental historical queries, a dynamic update mechanism and feedback mechanism are realized, which can learn user behavior and data changes in real time, continuously optimize the query matching and execution strategy, enable the system to have a high degree of intelligence and adaptability, be able to autonomously learn and optimize the query process, and provide intelligent data access and analysis capabilities.

[0096] Exemplarily, the server automatically generates description information for each database, which can be implemented in the following way:

[0097] For any database, use a description generation model to generate a description text of the database according to the historical queries and Data Definition Language (DDL) information of the database. This process can be expressed as: . Among them, represents the description text of the database. represents the DDL information of the database, including a set of SQL statements used to define and manage the database structure and objects. represents the historical queries of the database, including the historical natural language queries that match the database. LLM represents a large language model. Here, LLM is used as an example of the description generation model.

[0098] For any database, input the historical queries and DDL information of the database into the description generation model together, and generate a description text of the database through the description generation model. The description text summarizes information such as the structure, data objects, and uses of the database. Among them, the description generation model can be any large model, such as LLM or a deep learning model based on Transformer, which is not specifically limited here.

[0099] Furthermore, convert the description text of the database into a feature vector to obtain a description vector of the database. Exemplarily, input the description text of the database into a text encoding model for encoding, and the description text of the database can be converted into a high-dimensional vector representation. Use the high-dimensional vector representation of the description text of the database as the description vector of the database. Among them, the text encoding model can be implemented by any text encoder used to convert the input text into a vector representation, such as BERT (Bidirectional Encoder Representations from Transformers, a bidirectional encoder based on Transformer), recurrent neural network, convolutional neural network, etc., which is not specifically limited in this embodiment.

[0100] Optionally, the server automatically generates description information for each database, which can be implemented in the following way: For any database, input the historical queries and DDL information of the database into the LLM (including the backbone network and the output layer) together, and generate a description text of the database through the LLM. Use the hidden state vector output by the last layer of the backbone network of the LLM as the description vector of the database.

[0101] Exemplarily, the server automatically generates description information of the table connected subgraphs of each database, which can be implemented in the following manner:

[0102] For any database, according to the foreign key relationships between the data tables in the database, construct a foreign key (Foreign Key, abbreviated as FK) relationship graph of the data tables within the database. Among them, the foreign key relationship graph includes nodes and edges, where the nodes represent data tables and the edges represent the foreign key relationships between the data tables. Generate the maximum connected component (Maximum Connected Component, abbreviated as MCC) of the foreign key relationship graph to obtain the table connected subgraph of the database (denoted as ).

[0103] Furthermore, use a graph neural network (Graph Neural Network, abbreviated as GNN) to generate a description vector of the table connected subgraph of the database according to the historical queries and DDL information of the data tables included in the table connected subgraph of the database, and the foreign key relationships between the data tables included in the table connected subgraph of the database. Specifically, input the historical queries, the foreign key relationships between the first data tables, and the DDL information of the first data tables (each data table included in the table connected subgraph of the database is referred to as the first data table) into the graph neural network GNN for encoding, and output the description vector of the table connected subgraph of the database. This process can be expressed as: . Among them, represents the description vector of the table connected subgraph of the database. represents the historical queries of each data table (referred to as the first data table) included in the table connected subgraph of the database. represents the DDL information of each data table (referred to as the first data table) included in the table connected subgraph of the database. represents the foreign key relationships of each data table (referred to as the first data table) included in the table connected subgraph of the database.

[0104] Furthermore, input the description vector of the table connected subgraph of the database into a text generation model, and through the text generation model, generate the corresponding description text according to the description vector of the table connected subgraph of the database, then the description text of the table connected subgraph of the database can be obtained. Among them, the text generation model can be implemented using a fully connected layer, a long short-term memory (Long Short-Term Memory, abbreviated as LSTM) network, etc., and no specific limitation is made here in this embodiment.

[0105] In this embodiment, using a graph neural network (GNN) to vectorize the table connected subgraph can deeply understand the relationships between tables, improve the relevance of retrieving the target database and the natural language query, and thus improve the accuracy of retrieval.

[0106] Exemplarily, the server can automatically generate description information of data tables in each database, which can be implemented in the following way:

[0107] For any data table in any database, use a description generation model to generate a description text of the data table according to the historical queries and DDL information of the data table. This process can be expressed as: . Among them, represents the description text of the data table. represents the DDL information of the data table, including a set of SQL statements used to define and manage the data table structure and objects. represents the historical queries of the database, including the historical natural language queries that match the data table. LLM represents a large language model. Here, LLM is used as an example of the description generation model.

[0108] For any data table, input the historical queries and DDL information of the data table into the description generation model together, and generate a description text of the data table through the description generation model. The description text of the data table summarizes information such as the structure, data objects, and uses of the data table. Among them, the description generation model can be any large model, such as LLM or a deep learning model based on Transformer, which is not specifically limited here.

[0109] Furthermore, convert the description text of the data table into a feature vector to obtain the description vector of the data table. This process is the same as the implementation principle of converting the description text of the database into a feature vector to obtain the description vector of the database, which will not be elaborated here.

[0110] Optionally, the server can automatically generate description information of data table columns in each database, which can also be implemented in the following way: For any data table column in any data table of any database, input the historical queries and DDL information of the data table column into the LLM (including the backbone network and the output layer) together, and generate a description text of the data table column through the LLM. Use the hidden state vector output by the last layer of the backbone network of the LLM as the description vector of the data table column.

[0111] Exemplarily, the server can automatically generate description information of table columns in each database, which can be implemented in the following way:

[0112] For any table column in any data table of any database, use a description generation model to generate a description text of the table column according to the DDL information of the table column. This process can be expressed as: . Among them, represents the description text of the table column. represents the DDL information of the table column, including a set of SQL statements used to define and manage the table column structure and objects. LLM represents a large language model. Here, LLM is used as an example of the description generation model.

[0113] For any table column, input the DDL information of the table column into a description generation model, and generate the description text of the table column through the description generation model. The description text of the table column summarizes information such as the structure, data objects, and uses of the table column. Among them, the description generation model can be any large model, such as an LLM or a deep learning model based on Transformer, which is not specifically limited here.

[0114] Furthermore, convert the description text of the table column into a feature vector to obtain the description vector of the table column. The principle of this process is the same as that of converting the description text of the database into a feature vector to obtain the description vector of the database, which will not be elaborated here.

[0115] Optionally, the server automatically generates the description information of the table columns in each database, and can also be implemented in the following way: for any table column in any data table in any database, input the table column DDL information into an LLM (including the backbone network and the output layer), and generate the description text of the table column through the LLM. Use the hidden state vector output by the last layer of the backbone network of the LLM as the description vector of the table column.

[0116] It should be noted that when generating the description information of the database, the description information of the data table, and the description information of the table column, the same description generation model can be used, and the description vectors of the database, table, and table column can be mapped to a unified vector space to support complex query requirements.

[0117] It should be noted that this solution supports a fully automatic cold start mechanism. In the scenario of cold start (lack of historical queries), it is also possible to automatically generate the foregoing description information based on the DDL information of the database, data tables, and table columns, as well as the foreign key relationships of the data tables in the database.

[0118] The solution of this embodiment has the capabilities of fully automatic description information generation and dynamic update, ensuring that the description information always reflects the latest data usage and access patterns, greatly reducing the system's dependence on manual intervention, and improving the scalability and maintenance efficiency of the system. Moreover, by combining advanced vectorization technology with graph theory, constructing a graph structure through foreign key relationships, automatically identifying and generating the maximum connected component (MCC), it ensures that the relationships between tables are accurately captured and represented, and can achieve a deep understanding and efficient processing of the database structure and query intent.

[0119] In addition, based on the user's real-time query request, dynamic aggregation of incremental queries can be performed, and the database description information can be updated based on the incremental user queries. An incremental learning method can be used to continuously optimize the database description information to ensure that the system can adapt to changes in data and query requirements in real time.

[0120] Figure 4A flowchart of a rough recall candidate database provided for an exemplary embodiment of the present application. In an optional embodiment, in the rough recall stage, the server adopts a multi-path recall strategy, combining text matching and vector matching techniques, to quickly screen candidate databases related to natural language queries. Figure 4 As shown, in the aforementioned step S302, candidate databases related to the natural language query are recalled based on the description information of each database in the enterprise database and the description information of the table connected subgraph of each database, which can be specifically implemented in the following manner:

[0121] Step S401, based on the natural language query, the description text of each database and the description text of the table connected subgraph of each database, calculate the text similarity between the natural language query and each database, and screen out the first candidate set from which the candidate database comes according to the text similarity.

[0122] In this step, a recall strategy of text matching is adopted to quickly screen out databases related to the natural language query to form a first candidate set. Specifically, the server sorts each database to obtain a first sorting result according to the text similarity between the natural language query and the description text of each database and the description text of the table connected subgraph of each database, and then recalls the first candidate set according to the first sorting result.

[0123] Optionally, the server may calculate a first text similarity between the description text of each database and the natural language query, and / or a second text similarity between the description text of the table connected subgraph of each database and the natural language query. According to the first text similarity and / or the second text similarity, a first candidate set is determined. The first candidate set includes databases that meet the first screening condition.

[0124] The first screening condition includes: the first text similarity is greater than or equal to the first threshold, and / or the second text similarity is greater than or equal to the first threshold. The first threshold can be set according to actual application requirements or determined through experimental iteration, and is not specifically limited here.

[0125] Optionally, the server can also combine the first text similarity and the second text similarity to determine the comprehensive text similarity between the natural language query and each database; sort the databases in the first candidate set according to the comprehensive text similarity to obtain a first sorting result. According to the first sorting result, a second number of databases with top rankings are selected from the first candidate set as candidate databases. In the rough recall stage, a large number of candidate databases are recalled. Here, the second number can be specifically set according to the needs of the actual application scenario or determined through experimental iteration. For example, the second number can be set to 15, 20, etc., which is not specifically limited here. Among them, the comprehensive text similarity of the database can be the average of the first text similarity and the second text similarity corresponding to the database or the larger value of the two.

[0126] Step S402: Calculate the vector similarity between the natural language query and each database based on the vector representation of the natural language query, the description vectors of each database, and the description vectors of the table connectivity subgraphs of each database, and screen out the second candidate set from which the candidate databases come according to the vector similarity.

[0127] In this step, a recall strategy of vector matching is adopted to quickly screen out the databases related to the natural language query and form the second candidate set.

[0128] Specifically, the server sorts each database according to the vector similarity between the vector representation of the natural language query and the description vectors of each database and the description vectors of the table connectivity subgraphs of each database to obtain the second sorting result, and recalls the second candidate set according to the second sorting result.

[0129] Exemplarily, the server converts the natural language query into a vector representation, calculates the first vector similarity between the vector representation and the description vectors of each database, and / or the second vector similarity between the vector representation and the description vectors of the table connectivity subgraphs of each database. Determine the second candidate set according to the first vector similarity and / or the second vector similarity. The second candidate set includes databases that meet the second screening condition.

[0130] Among them, the second screening condition includes: the first vector similarity is greater than or equal to the second threshold, and / or the second vector similarity is greater than or equal to the second threshold. The second threshold can be set according to actual application requirements or determined through experimental iteration, and no specific limitation is made here.

[0131] Optionally, the server can also comprehensively consider the first vector similarity and the second vector similarity to determine the comprehensive vector similarity between the natural language query and each database; sort the databases in the second candidate set according to the comprehensive vector similarity to obtain the second sorting result. According to the second sorting result, select the second number of databases with higher rankings from the second candidate set as the candidate databases. Recall a larger number of candidate databases in the rough recall stage. Here, the second number can be specifically set according to the needs of the actual application scenario or determined through experimental iteration. For example, the second number can be set to 15, 20, etc., and no specific limitation is made here. Among them, the comprehensive vector similarity of the database can be the mean value of the first vector similarity and the second vector similarity corresponding to the database or the larger value of the two.

[0132] Step S403: Calculate the comprehensive similarity between the natural language query and each database according to the text similarity and the vector similarity.

[0133] In this step, by comprehensively considering the text similarity and the vector similarity corresponding to each database, the comprehensive similarity between the natural language query and each database can be determined.

[0134] Optionally, based on the comprehensive text similarity and comprehensive vector similarity corresponding to each database calculated in the foregoing steps S401 - S402, the comprehensive text similarity and comprehensive vector similarity of the same database can be weighted and summed (or weighted averaged) to obtain the comprehensive similarity corresponding to the database, that is, the comprehensive similarity between the natural language query and each database. Among them, the weight coefficients of the comprehensive text similarity and comprehensive vector similarity can be set according to the actual application scenario, and no specific limitation is made here.

[0135] Optionally, based on the first text similarity, second text similarity, first vector similarity, and second vector similarity corresponding to each database calculated in the foregoing steps S401 - S402, the first text similarity, second text similarity, first vector similarity, and second vector similarity of the same database can be weighted and summed (or weighted averaged) to obtain the comprehensive similarity corresponding to the database. Among them, the weight coefficients of the first text similarity, second text similarity, first vector similarity, and second vector similarity can be set according to the actual application scenario, and no specific limitation is made here.

[0136] It should be noted that in this step, only the databases in the first candidate set and the second candidate set determined in the foregoing steps S401 - S402 need to be calculated for comprehensive similarity and sorted, while other databases that have been filtered out are no longer calculated, which can save computing resources and improve the efficiency of data processing.

[0137] Step S404: Recall candidate databases from the first candidate set and the second candidate set according to the comprehensive similarity.

[0138] After calculating the comprehensive similarity corresponding to each database in the first candidate set and the second candidate set, sort the databases in the first candidate set and the second candidate set according to the comprehensive similarity to obtain the third sorting result. Screen out the top sixth number of databases according to the third sorting result to obtain the roughly recalled candidate databases.

[0139] Among them, the sixth number can be specifically set according to the needs of the actual application scenario or determined through experimental iteration. For example, the sixth number can be set to 20, 30, etc. No specific limitation is made here.

[0140] In this embodiment, in the rough recall stage, a multi - path recall strategy is adopted, combining text matching and vector matching technologies. By combining multi - path recall strategies such as vector matching, connected sub - graph expansion, and text matching, candidate databases related to the natural language query can be quickly screened, which can cover a wider range of data sources and query requirements, enhance the query coverage rate, ensure a high - coverage query result, and meet the diverse needs of enterprise users.

[0141] Figure 5 The flowchart of fine sorting provided by an exemplary embodiment of the present application. In an alternative embodiment, based on the candidate database obtained by rough recall, in the fine sorting stage, the RRF method is adopted to comprehensively calculate the RRF score according to the rankings of the candidate database in different recall strategies, and combined with the table priority, table vector similarity, and table column vector similarity, the candidate database obtained by rough recall is accurately sorted, and the accurate sorting of the rough recall result is realized through a hierarchical fine sorting logic, so as to improve the positioning accuracy of the target database, target data table, and target table column associated with the natural language query, and improve the relevance and accuracy of the recalled target results (including the target database, target data table, and target table column).

[0142] As Figure 5 shown, in the foregoing step S303, according to the natural language query, the description information of each data table in the candidate database, and the description information of each table column, the candidate database is finely sorted to obtain a fine sorting result, which can be specifically implemented in the following manner:

[0143] Step S501: Calculate at least one fine sorting score of the candidate database according to the natural language query, the description information of each data table in the candidate database, and the description information of each table column.

[0144] In this embodiment, a multi-level fine sorting logic of multi-strategy fusion is adopted in the fine sorting stage. For any candidate database, the server can calculate the following at least one fine sorting score of the candidate database: reciprocal rank fusion RRF score, table vector similarity score, table column vector similarity score, table priority score.

[0145] Specifically, for the RRF score of any candidate database, the server can calculate the reciprocal rank fusion RRF score of the candidate database according to the recall ranking of the candidate database in the rough recall stage, denoted as .

[0146] Exemplarily, for the RRF score of the candidate database, the server can calculate the RRF score of the candidate database according to the recall rankings corresponding to different recall strategies of the candidate database in the rough recall stage. This process can be expressed as: . Among them, represents the number of different recall strategies used in the rough recall stage (i.e., the number of recall paths). represents the ranking of the candidate database in the sorting result of the i-th recall strategy, that is, the recall ranking corresponding to the i-th recall strategy of the candidate database in the rough recall stage. is a tuning parameter, which is a small constant, for example, it can take a value of 0.0001.

[0147] For example, taking Figure 4Taking the rough recall process provided by the corresponding embodiment as an example, in the rough recall stage, two recall strategies of text matching and vector matching are adopted, corresponding to the first sorting result and the second sorting result respectively. For any candidate database, the ranking of the candidate database in the first sorting result can be based on and the ranking of the candidate database in the second sorting result to calculate the RRF score of the candidate database.

[0148] Optionally, for any candidate database, the server can calculate the reciprocal of the ranking of the candidate database in the third sorting result in the rough recall stage as the RRF score of the candidate database. Optionally, the server can comprehensively consider the rankings of the candidate database in the first sorting result, the second sorting result, and the third sorting result in the rough recall stage, and regard the third sorting result as the sorting result of a recall strategy (at this time k = 3) to calculate the RRF score of the candidate database.

[0149] Exemplarily, for the table vector similarity score of any candidate database, the server can calculate the table vector similarity score of the candidate database according to the third vector similarity between the vector representation of the natural language query and the description vectors of each data table in the candidate database.

[0150] Optionally, the server can filter out the data tables with the third vector similarity greater than the seventh threshold according to the third vector similarity between the vector representation of the natural language query and the description vectors of each data table in the candidate database, and calculate the average value of the third vector similarities of the data tables with the third vector similarity greater than the seventh threshold to obtain the table vector similarity score of the candidate database, denoted as . This process can be expressed as: . Among them, represents the seventh threshold, which can be determined according to the actual application requirements and experimental iterations, and is not specifically limited here. represents the number of data tables with the third vector similarity greater than the seventh threshold. represents the third vector similarity between the vector representation of the natural language query and the description vector of any data table.

[0151] Optionally, the server can calculate the maximum value of the third vector similarities between the vector representation of the natural language query and the description vectors of each data table in the candidate database as the table vector similarity score of the candidate database.

[0152] Exemplarily, for the table column vector similarity score of any candidate database, the server can calculate the table column vector similarity score of the candidate database according to the fourth vector similarity between the vector representation of the natural language query and the description vectors of each table column in the candidate database.

[0153] Optionally, the server can screen out the table columns whose fourth vector similarity is greater than the eighth threshold according to the fourth vector similarity between the vector representation of the natural language query and the description vectors of each table column in the candidate database, calculate the average value of the fourth vector similarities of the table columns whose fourth vector similarity is greater than the eighth threshold, obtain the table column vector similarity score of the candidate database, denoted as . This process can be expressed as: . Among them, represents the eighth threshold, which can be determined according to actual application requirements and experimental iterations, and is not specifically limited here. represents the number of table columns whose fourth vector similarity is greater than the eighth threshold. represents the fourth vector similarity between the vector representation of the natural language query and the description vector of any table column.

[0154] Optionally, the server can calculate the maximum value of the fourth vector similarities between the vector representation of the natural language query and the description vectors of each table column in the candidate database as the table column vector similarity score of the candidate database.

[0155] Exemplarily, for the table priority score of any candidate database, the server can calculate the table priority score of the candidate database according to the historical query frequencies of each data table in the candidate database, denoted as .

[0156] Optionally, the server can screen out the data tables whose third vector similarity is greater than the seventh threshold according to the third vector similarity between the vector representation of the natural language query and the description vectors of each data table in the candidate database. For the data tables whose third vector similarity is greater than the seventh threshold, calculate the ratio of the historical query frequency of each data table to the total query frequency of the candidate database as the score corresponding to each data table. Calculate the average value of the scores corresponding to each data table whose third vector similarity is greater than the seventh threshold to obtain the table priority score of the candidate database.

[0157] Optionally, the server can sort the data tables in the candidate database according to the third vector similarity between the vector representation of the natural language query and the description vectors of each data table in the candidate database, screen out the seventh number of data tables with the top ranking. Calculate the ratio of the historical query frequency of each data table to the total query frequency of the candidate database as the score corresponding to each data table. Calculate the average value of the scores corresponding to the seventh number of data tables with the top ranking to obtain the table priority score of the candidate database.

[0158] Step S502: Determine the comprehensive ranking score of the candidate database according to at least one ranking score of the candidate database.

[0159] After calculating at least one fine-ranking score of the candidate databases, the server sums up the weighted fine-ranking scores of each candidate database to obtain the comprehensive fine-ranking score of the candidate databases, denoted as . The weight coefficients of each fine-ranking score can be set and adjusted according to actual application requirements, and no specific limitation is made here.

[0160] Exemplarily, the comprehensive fine-ranking score of the candidate databases can be calculated in the following way: . Among them, , is the weight coefficient.

[0161] Step S503: Sort the candidate databases according to the comprehensive fine-ranking scores of the candidate databases to obtain the fine-ranking result.

[0162] After obtaining the comprehensive fine-ranking scores of each candidate database, sort the candidate databases according to the comprehensive fine-ranking scores of the candidate databases to obtain the fine-ranking result.

[0163] In the solution of this embodiment, based on the candidate databases obtained by rough recall, in the fine-ranking stage, the RRF method is adopted to comprehensively calculate the RRF scores based on the rankings of the candidate databases in different recall strategies, and combined with the table priority, table vector similarity, and table column vector similarity, calculate multiple fine-ranking scores of the candidate databases, and comprehensively sort the candidate databases obtained by rough recall according to the multiple fine-ranking scores, and realize the accurate ranking of the rough recall results through the hierarchical fine-ranking logic, so as to improve the relevance and accuracy of the target results to be recalled (including the target database, target data table, and target table column).

[0164] Exemplarily, Figure 6 is an example framework diagram of the intelligent routing provided by the embodiment of the present application. As Figure 6 shown, based on the intelligent routing strategy provided by the present application, for the input natural language query (i.e., user Query), in the rough recall stage, multiple different recall strategies (such as vector recall and ES recall shown in the figure) are adopted for multi-path recall, and the candidate databases related to the user Query are roughly screened out. On the basis of rough recall, a multi-level fine-ranking strategy (such as RRF fine-ranking, table column recall fine-ranking, and vector fine-ranking shown in the figure) is adopted to calculate multiple fine-ranking scores of the candidate databases, including but not limited to RRF scores, table vector similarity scores, table column vector similarity scores, and table priority scores. Comprehensively sort the candidate databases obtained by rough recall according to the multiple fine-ranking scores, and screen out the target database, target data table, and target table column associated with the user Query according to the fine-ranking result to obtain the final recall result of the database and table column.

[0165] In addition, as Figure 6As shown, the server can automatically generate the description information of the unified database, the description information of the table connection subgraph of the database, the description information of the data table, and the description information of the table column, providing a data basis for the rough recall and fine sorting stages. And as the user queries continue to increase, the server can dynamically update the description information of the database, the description information of the table connection subgraph of the database, the description information of the data table, and the description information of the table column. The generation and dynamic update process of the description information are fully automated without manual intervention, significantly improving the data processing efficiency and accuracy.

[0166] Figure 7 The flowchart of the data processing method provided by another exemplary embodiment of the present application. The execution subject of this embodiment can be a server that implements NL2SQL. As Figure 7 shown, the specific steps of this method are as follows:

[0167] Step S701: In response to a natural language to SQL conversion request, obtain the natural language query to be converted.

[0168] In the ChatBI query scenario based on the enterprise-level database, the user can send a natural language to SQL conversion request to the server through the end-side device, and the request carries the natural language query input by the user. The server receives the natural language to SQL conversion request sent by the end-side device and obtains the natural language query.

[0169] Exemplarily, the natural language to SQL conversion request can be a call request from the end-side device to the NL2SQL service API. The server externally provides the API (Application Programming Interface) of the NL2SQL service to the end-side device. When the end-side device needs to use the NL2SQL service, it takes the natural language query input by the user as an input parameter and sends a call request for the API of the NL2SQL service to the server. In response to receiving the call request for the API of the NL2SQL service, the server obtains the natural language query as the input parameter.

[0170] Step S702: Recall candidate databases related to the natural language query according to the description information of each database and the description information of the table connection subgraph of each database.

[0171] Step S703: Fine-sort the candidate databases according to the natural language query, the description information of each data table in the candidate database, and the description information of each table column, and obtain a fine-sorting result.

[0172] Step S704: Select at least one candidate database according to the fine-sorting result as the target database associated with the natural language query.

[0173] Step S705: Determine the target data table and target table column associated with the natural language query in the target database.

[0174] In this embodiment, the implementation principles of steps S702 - S705 are the same as those of the foregoing steps S302 - S305. For specific details, refer to the relevant content in the foregoing embodiment, and details will not be elaborated here.

[0175] Step S706: Generate an SQL statement corresponding to the natural language query according to the target database, target data table, and target table column.

[0176] In this embodiment, after quickly locating the target database, target data table, and target table column associated with the natural language query through a unified intelligent routing strategy based on the input natural language query, the server can also generate a structured query language SQL statement corresponding to the natural language query according to the target database, target data table, and target table column, realizing the NL2SQL function.

[0177] For any target database, the server converts the natural language query into an SQL statement that can be executed in the target database according to the target data table and target table column in the target database. Specifically, it can be implemented by any method for realizing NL2SQL for a single application-level database, and details will not be elaborated here.

[0178] Step S707: Output the SQL statement.

[0179] In this step, the server returns the generated SQL statement to the terminal device. The terminal device can execute the corresponding SQL statement in the target database to obtain the data query result.

[0180] The solution of this embodiment, in the face of multiple databases and a large number of data tables, based on an intelligent routing strategy, within the global scope of the database, based on the unified description information of the database, the description information of the table connectivity subgraph of the database, the description information of the data table, and the description information of the table column, intelligently routes the user query to the target database, target data table, and target table column associated with the user query within the global scope, and generates an SQL statement corresponding to the natural language query according to the target database, target data table, and target table column, realizing the NL2SQL processing for the user's global data query requirements, and without relying on complex data governance and scenario isolation, simplifies the multi-database query process through unified intelligent routing, and improves the operation convenience of multi-database query.

[0181] Figure 8 It is a schematic structural diagram of a server provided by an embodiment of the present application. As Figure 8As shown in the figure, the server includes: a memory 801 and a processor 802. The memory 801 is used to store computer-executable instructions and can be configured to store various other data to support operations on the server. The processor 802 is communicatively connected to the memory 801 and is used to execute the computer-executable instructions stored in the memory 801 to implement the technical solutions provided in any of the foregoing method embodiments. Its specific functions and achievable technical effects are similar and will not be elaborated herein.

[0182] Optionally, as Figure 8 shown in the figure, the server further includes: other components such as a firewall 803, a load balancer 804, a communication component 805, a power supply component 806, etc. Figure 8 Only some components are schematically shown in the figure, which does not mean that the server only includes Figure 8 the components shown in the figure. Figure 8 Only a cloud server deployed in the cloud is taken as an example for illustrative purposes. The server can also be deployed locally, and no specific limitation is made here in this embodiment.

[0183] An embodiment of the present application also provides a computer-readable storage medium. Computer-executable instructions are stored in the computer-readable storage medium. When the processor executes the computer-executable instructions, the method of any of the foregoing embodiments is implemented. Its specific functions and achievable technical effects will not be elaborated herein.

[0184] An embodiment of the present application also provides a computer program product, including a computer program. When the computer program is executed by the processor, the method of any of the foregoing embodiments is implemented. The computer program is stored in a readable storage medium. At least one processor of the server can read the computer program from the readable storage medium, and at least one processor executes the computer program to enable the server to execute the technical solutions provided in any of the foregoing method embodiments. Its specific functions and achievable technical effects will not be elaborated herein.

[0185] An embodiment of the present application provides a chip, including: a processing module and a communication interface. The processing module can execute the technical solutions of the server in any of the foregoing method embodiments. Optionally, the chip further includes a storage module (such as a memory). The storage module is used to store instructions, and the processing module is used to execute the instructions stored in the storage module. And the execution of the instructions stored in the storage module enables the processing module to execute the technical solutions provided in any of the foregoing method embodiments.

[0186] The integrated modules implemented in the form of software function modules as described above can be stored in a computer-readable storage medium. The above software function modules are stored in a storage medium and include several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute some steps of the methods in various embodiments of the present application.

[0187] It should be understood that the above-mentioned processor may be a central processing unit (CPU), a graphics processing unit (GPU), or other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the application can be directly embodied as being executed and completed by a hardware processor, or executed and completed by a combination of hardware and software modules in at least one processor.

[0188] The memory may include high-speed random access memory (RAM), and may also include non-volatile storage, such as at least one disk memory, and may also be a USB flash drive, a portable hard drive, a read-only memory, a magnetic disk, or an optical disc, etc.

[0189] The above-mentioned memory may be an object storage service (OSS). The above-mentioned memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disc.

[0190] The above communication component is configured to facilitate communication, either wired or wireless, between the device where the communication component is located and other devices. The device where the communication component is located can access a communication standard-based wireless network, such as a mobile hotspot (WiFi), a second-generation mobile communication system (2G), a third-generation mobile communication system (3G), a fourth-generation mobile communication system (4G) / Long Term Evolution (LTE), a fifth-generation mobile communication system (5G), etc., or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, infrared technology, Ultra Wide Band (UWB) technology, Bluetooth technology, and other technologies.

[0191] The above power supply component provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0192] The above storage medium can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0193] An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application-specific integrated circuit. Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a master device.

[0194] It should be noted that in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or device including that element.

[0195] The order of the above embodiments of the present application is only for description and does not represent the superiority or inferiority of the embodiments. Additionally, in some of the processes described in the above embodiments and the accompanying drawings, there are multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order in which they appear in this article or may be executed in parallel, and are only used to distinguish different operations. The serial numbers themselves do not represent any order of execution. Additionally, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions such as "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., do not represent a sequence, and do not limit that "first" and "second" are of different types. The meaning of "a plurality" is more than two, unless otherwise specifically and clearly defined.

[0196] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc) and includes several instructions to enable a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of the present application.

[0197] After considering the specification and practicing the invention disclosed herein, those skilled in the art will readily conceive of other embodiments of the present application. The present application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common general knowledge or conventional technical means in the technical field not disclosed in the present application.

[0198] The above are only the preferred embodiments of the present application, and do not limit the patent scope of the present application accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be similarly included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: include: Receiving an input natural language query; Recalling candidate databases related to the natural language query according to the description information of each database and the description information of the table connected subgraph of each database; According to the natural language query, description information of each data table and description information of each table column in the candidate database, the candidate database is finely sorted to obtain a finely sorted result; Selecting at least one of the candidate databases according to the refined ranking result as a target database associated with the natural language query; A target data table and a target table column associated with the natural language query are determined in the target database.

2. The method according to claim 1, characterized in that: Also includes: Generate and regularly update description information of each database, description information of table connected subgraphs of each database, description information of data tables in each database, and description information of table columns in each database; The description information of the database includes: description text and / or description vector of the database; The description information of the table connected subgraph of the database includes: a description text of the table connected subgraph and / or a description vector of the table connected subgraph; The description information of the data table includes: a description text of the data table and / or a description vector of the data table; The description information of the table column includes: a description text of the table column and / or a description vector of the table column.

3. The method according to claim 2, characterized in that Generate description information of each database, including: For any of the databases, using a description generation model, generating a description text of the database according to historical queries and data definition language DDL information of the database; The description text of the database is converted into a feature vector to obtain a description vector of the database.

4. The method according to claim 2, characterized in that: Generate description information of the table connected subgraphs of the databases, including: For any of the databases, a foreign key relationship graph of the data tables in the database is constructed according to the foreign key relationships between the data tables in the database, wherein the foreign key relationship graph includes nodes and edges, the nodes represent the data tables, and the edges represent the foreign key relationships between the data tables; Generate a maximum connected subgraph of the foreign key relationship graph to obtain a table connected subgraph of the database; Using a graph neural network, generating a description vector of the table connected subgraph of the database according to historical queries, foreign key relationships, and DDL information of data tables contained in the table connected subgraph of the database; A description text of the table connected subgraph of the database is generated according to the description vector of the table connected subgraph of the database through a text generation model.

5. The method according to claim 2, characterized in that: Generate description information of the data tables in each database, including: For any data table in any of the databases, using a description generation model, a description text of the data table is generated according to historical queries and DDL information of the data table; The description text of the data table is converted into a feature vector to obtain a description vector of the data table.

6. The method according to claim 2, characterized in that Generate description information of the table columns in each database, including: For any table column in any data table of any of the databases, using the description generation model, according to the DDL information of the table column, a description text of the table column is generated; The description text of the table column is converted into a feature vector to obtain the description vector of the table column.

7. The method according to any one of claims 2 to 6, characterized in that: The recalling of candidate databases related to the natural language query according to the description information of each database and the description information of the table connected subgraph of each database includes: Calculating text similarity between the natural language query and each of the databases based on the natural language query, description text of each of the databases, and description text of the table connected subgraph of each of the databases, and filtering out a first candidate set from which the candidate database comes based on the text similarity; Calculating the vector similarity between the natural language query and each of the databases according to the vector representation of the natural language query, the description vector of each of the databases, and the description vector of the table connected subgraph of each of the databases, and filtering out the second candidate set from which the candidate database comes according to the vector similarity; Calculating the comprehensive similarity between the natural language query and each of the databases according to the text similarity and the vector similarity; The candidate database is screened out from the first candidate set and the second candidate set according to the comprehensive similarity.

8. The method according to claim 7, characterized in that The step of calculating text similarity between the natural language query and each of the databases based on the natural language query, the description text of each of the databases, and the description text of the table connected subgraph of each of the databases, and selecting a first candidate set of the candidate databases based on the text similarity, comprises: Calculating a first text similarity between the description text of each of the databases and the natural language query, and / or a second text similarity between the description text of the table connected subgraph of each of the databases and the natural language query; The first candidate set is determined according to the first text similarity and / or the second text similarity, and the first candidate set includes a database that meets a first screening condition; wherein the first screening condition includes: the first text similarity is greater than or equal to a first threshold, and / or the second text similarity is greater than or equal to the first threshold.

9. The method according to claim 7, characterized in that: The calculating the vector similarity between the natural language query and each of the databases according to the vector representation of the natural language query, the description vector of each of the databases, and the description vector of the table connected subgraph of each of the databases, and screening out the second candidate set of the candidate databases according to the vector similarity includes: Converting the natural language query into a vector representation; Calculating the similarity between the description vector of each of the databases and the first vector represented by the vector, and / or the similarity between the description vector of the table connected subgraph of each of the databases and the second vector represented by the vector; The second candidate set is determined based on the first vector similarity and / or the second vector similarity, and the second candidate set includes a database that meets a second filtering condition; wherein the second filtering condition includes: the first vector similarity is greater than or equal to a second threshold, and / or the second vector similarity is greater than or equal to the second threshold.

10. The method according to any one of claims 2 to 6, characterized in that: The step of finely sorting the candidate database according to the natural language query, the description information of each data table and the description information of each table column in the candidate database to obtain a fine sorting result includes: Calculate at least one refined ranking score of the candidate database according to the natural language query, description information of each data table and description information of each table column in the candidate database; Determining a comprehensive refined ranking score of the candidate database according to the at least one refined ranking score of the candidate database; The candidate databases are sorted according to their comprehensive refined ranking scores to obtain refined ranking results.

11. The method according to claim 10, characterized in that The step of calculating at least one refined ranking score of the candidate database according to the natural language query, the description information of each data table and the description information of each table column in the candidate database comprises: For any of the candidate databases, calculate at least one of the following refined ranking scores of the candidate database: Calculating a reciprocal ranking fusion score of the candidate database according to the recall ranking of the candidate database; Calculating a table vector similarity score of the candidate database according to a third vector similarity between the vector representation of the natural language query and the description vector of each data table in the candidate database; Calculating a vector similarity score of the table columns of the candidate database according to a fourth vector similarity between the vector representation of the natural language query and the description vector of each table column in the candidate database; The table priority score of the candidate database is calculated according to the historical query frequency of each data table in the candidate database.

12. The method according to any one of claims 1 to 6, characterized in that Also includes: Generate a structured query language SQL statement corresponding to the natural language query according to the target database, target data table and target table column; Output the SQL statement.

13. The method according to any one of claims 1 to 6, characterized in that Determining a target data table and a target table column associated with the natural language query in the target database includes: Determining a target data table in the target database according to a matching degree between the natural language query and description information of each data table in the target database; According to the matching degree between the natural language query and the description information of each table column in the target data table, the target table column in the target data table is determined.

14. A data processing method, characterized in that: include: In response to a natural language to SQL conversion request, obtaining a natural language query to be converted; Recalling candidate databases related to the natural language query according to the description information of each database and the description information of the table connected subgraph of each database; According to the natural language query, description information of each data table and description information of each table column in the candidate database, the candidate database is finely sorted to obtain a finely sorted result; Selecting at least one of the candidate databases according to the refined ranking result as a target database associated with the natural language query; Determine in the target database a target data table and a target table column associated with the natural language query; Generate an SQL statement corresponding to the natural language query according to the target database, target data table and target table column; Output the SQL statement.

15. A server, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the server to execute the method described in any one of claims 1-14.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 14 is implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images