Structured query language statement generation method and related device

Abstract

A structured query language statement generation method, executed by an electronic device, and comprising: determining a natural language text representing a query task, and determining a preset database, the preset database comprising at least one data table, and each data table comprising at least one field (101); on the basis of the natural language text, determining, from the preset database, at least one task-associated field and at least one task-associated data table which are associated with the natural language text (102); rewriting the natural language text on the basis of the task-associated field to obtain a rewritten natural language text (103); acquiring prompt information for the rewritten natural language text, the prompt information being used for guiding a query statement generation model in generating a structured query language statement (104); performing question classification on the rewritten natural language text to obtain question type information of the rewritten natural language text (105); and generating the structured query language statement of the rewritten natural language text by means of the query statement generation model on the basis of the task-associated field, the task-associated data table, the prompt information, and the question type information (106).

Description

Structured query statement generation method and related device

[0001] Related applications

[0002] The present application claims priority to the Chinese patent application No. 202411071596.4, filed on August 5, 2024, and entitled "Structured query statement generation method and related device", the contents of which are hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of computers, in particular to a structured query statement generation method and related device. BACKGROUND

[0004] Natural Language to SQL (NL2SQL) is a type of semantic parsing task, which aims to convert the natural language question input by the user into a structured query language (SQL) that can operate with the database, so as to realize the interaction with the database through the natural language and obtain the information in the database.

[0005] In the related art, the NL2SQL method is usually to generate the corresponding SQL statement for the natural language question input by the user through the trained neural network model. Since the SQL generation method is relatively rough, it only relies on the activation ability of the neural network model, and the noise resistance is insufficient, and the comprehensive consideration of the SQL generation task is lacking, which is not conducive to improving the accuracy of the SQL query generation. SUMMARY

[0006] The present application provides a structured query statement generation method and related device.

[0007] The present application provides a structured query statement generation method, which is executed by an electronic device, comprising:

[0008] determining a natural language text representing a query task, and determining a preset database, the preset database comprising at least one data table, and each data table comprising at least one field;

[0009] determining at least one task-associated field and at least one task-associated data table associated with the natural language text from the preset database based on the natural language text;

[0010] rewriting the natural language text according to the task-associated field to obtain a rewritten natural language text;

[0011] obtain prompt information for the rewritten natural language text, the prompt information being used to guide a query statement generation model to generate a structured query statement;

[0012] perform question classification on the rewritten natural language text to obtain question type information of the rewritten natural language text; and

[0013] generate, by the query statement generation model, a structured query statement of the rewritten natural language text based on the task-associated field, the task-associated data table, the prompt information, and the question type information.

[0014] Correspondingly, an embodiment of the present application provides a structured query statement generation apparatus, comprising:

[0015] a first determination unit configured to determine natural language text representing a query task and determine a preset database, the preset database comprising at least one data table, and each data table comprising at least one field;

[0016] a second determination unit configured to determine, based on the natural language text, at least one task-associated field and at least one task-associated data table associated with the natural language text from the preset database;

[0017] a rewriting unit configured to rewrite the natural language text according to the task-associated field to obtain rewritten natural language text;

[0018] an obtaining unit configured to obtain prompt information for the rewritten natural language text, the prompt information being used to guide a query statement generation model to generate a structured query statement;

[0019] a question classification unit configured to perform question classification on the rewritten natural language text to obtain question type information of the rewritten natural language text; and

[0020] a generation unit configured to generate, by the query statement generation model, a structured query statement of the rewritten natural language text based on the task-associated field, the task-associated data table, the prompt information, and the question type information.

[0021] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the function of the module or unit.

[0022] An electronic device provided by an embodiment of the present application includes a processor and a memory. The memory stores a plurality of instructions. The processor loads the instructions to perform steps in the structured query statement generation method provided by an embodiment of the present application.

[0023] An embodiment of the present application further provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, steps in the structured query statement generation method provided by an embodiment of the present application are implemented.

[0024] In addition, an embodiment of the present application further provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, steps in the structured query statement generation method provided by an embodiment of the present application are implemented.

[0025] Details of one or more embodiments of the present application are presented in the following drawings and description. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of the disclosed drawings.

[0027] FIG. 1a is a schematic diagram of a scenario of the structured query statement generation method provided by an embodiment of the present application;

[0028] FIG. 1b is a flowchart of the structured query statement generation method provided by an embodiment of the present application;

[0029] FIG. 1c is an explanatory diagram of the structured query statement generation method provided by an embodiment of the present application;

[0030] FIG. 1d is a model architecture diagram of the structured query statement generation method provided by an embodiment of the present application;

[0031] FIG. 1e is another explanatory diagram of the structured query statement generation method provided by an embodiment of the present application;

[0032] FIG. 2 is another flowchart of the structured query statement generation method provided by an embodiment of the present application;

[0033] FIG. 3 is a structural schematic diagram of a structured query statement generation apparatus provided by an embodiment of the present application;

[0034] FIG. 4 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0036] The embodiments of the present application provide a structured query statement generation method and related devices. The related devices can include a structured query statement generation apparatus, an electronic device, a computer readable storage medium, and a computer program product. The structured query statement generation apparatus can be integrated in an electronic device, which can be a terminal or a server.

[0037] It can be understood that the structured query statement generation method of the present embodiment can be executed on a terminal, on a server, or by a terminal and a server together. The above examples should not be understood as limiting the present application.

[0038] As shown in FIG. 1a, taking the terminal and the server together to execute the structured query statement generation method as an example. The structured query statement generation system provided by the embodiments of the present application includes a terminal 10 and a server 11, etc. The terminal 10 and the server 11 are connected through a network, such as a wired or wireless network connection, etc. The structured query statement generation apparatus can be integrated in the server.

[0039] The server 11 can be configured to determine a natural language text representing a query task, and determine a preset database. The preset database includes at least one data table, and each data table includes at least one field. The server 11 can be configured to determine at least one task-associated field and at least one task-associated data table associated with the natural language text from the preset database based on the natural language text. The server 11 can be configured to rewrite the natural language text according to the task-associated field to obtain a rewritten natural language text. The server 11 can be configured to obtain prompt information for the rewritten natural language text, the prompt information being used to guide a query statement generation model to generate a structured query statement. The server 11 can be configured to perform question classification on the rewritten natural language text to obtain question type information of the rewritten natural language text. The server 11 can be configured to generate a structured query statement of the rewritten natural language text based on the task-associated field, the task-associated data table, the prompt information, and the question type information through the query statement generation model. The server 11 can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0040] The terminal 10 can be configured to obtain the natural language text representing the query task and send the natural language text representing the query task to the server 11. The terminal 10 can also receive the structured query statement of the rewritten natural language text sent by the server 11. The terminal 10 can include a mobile phone, a vehicle-mounted terminal, an aircraft, a tablet computer, a notebook computer, or a personal computer (PC), etc. The terminal 10 can also be provided with a client, which can be an application program client or a browser client, etc.

[0041] The server 11 can also perform the steps of generating the structured query statement, etc.

[0042] The following will be described in detail. It should be noted that the order of the following embodiments is not limited as the preferred order of the embodiments.

[0043] The embodiment will be described from the perspective of a structured query statement generation device, which can be integrated in an electronic device, such as a server or a terminal, etc.

[0044] It can be understood that in the specific embodiments of the present application, the data related to user information is involved. When the above embodiments of the present application are applied to specific products or technologies, the user's permission or consent is required, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of the country and region.

[0045] As shown in FIG. 1b, the specific process of the structured query statement generation method can be as follows:

[0046] 101. Determine the natural language text representing the query task and determine the preset database, the preset database including at least one data table, each data table including at least one field.

[0047] The natural language text is specifically a natural language question text proposed by a user. The preset database is an electronic warehouse for organizing, storing and managing related business data. The user can perform operations such as adding, querying, updating and deleting on the data in the preset database.

[0048] The preset database can be a preset unique database. The preset database can also be determined by a system configuration file and user permission information after the natural language text representing the query task is determined. Specifically, after the natural language text representing the query task is determined, a system configuration file is first read, in which a list of databases corresponding to different business scenarios is pre-stored. Then, according to the business requirements and operation permissions of the user, the databases that meet the user's permissions are filtered from the database list of the configuration file. For example, if the user's permissions only allow access to data of a specific business line, the database corresponding to the business line is filtered from the list. At the same time, the database metadata management system is used to perform health check and data integrity verification on the filtered databases, to check whether the databases are in normal operation state, and whether the definitions of data tables and fields are complete and conform to the specifications, to ensure the availability and accuracy of the preset database, and finally determine the databases that pass the verification as the preset database.

[0049] In specific business scenarios, with the explosive growth of enterprise data, data analysis plays an increasingly important role in enterprise decision-making, product optimization, and operation management. However, data analysis often requires certain technical capabilities, such as proficiency in SQL (Structured Query Language) programming language and familiarity with various table information. However, understanding thousands of tables requires a lot of time and effort, which puts an additional burden on product operators and data analysts. To address this issue, Text2SQL applications can be used to lower the technical threshold. Text2SQL applications use natural language processing technology to quickly convert user text into SQL query statements and data analysis results, avoiding the need for users to spend more time understanding the contents of related table fields, thereby improving work efficiency and saving time and effort.

[0050] In the current related technology, a high-level model (such as GPT) is generally directly used to implement the text2sql task. However, in the face of complex business data scenarios, this method has the following disadvantages: first, data privacy and security issues, calling GPT and other models can cause data leakage, and data security is a red line for enterprises; second, existing large models do not understand business data well, the data quality of specific businesses is low, the table data is messy, and the table creation habits of each staff member are different, making it more difficult for the model to understand these data; third, stability and accuracy are not enough, specifically, real users' question asking methods are very personalized, and the existing method lacks noise resistance, resulting in poor stability and accuracy; fourth, there is a lack of high-quality data, specifically, in the initial cold start stage, there is a lack of high-quality query-sql data, and it is difficult to construct SQL data.

[0051] The structured query statement generation method provided by the application can train a self-owned model to avoid calling a GPT model, solve the problem of data privacy, and improve data security. Moreover, the application also designs a complete process of an agent (intelligent agent) to greatly improve the accuracy of SQL generation, thereby providing more efficient and accurate data support for data analysis. Specifically, the structured query statement generation method of the application can be applied to various scenarios requiring data analysis, such as specific business scenarios such as news information, video apps, browsers, and the like.

[0052] 102. Determine at least one task-associated field and at least one task-associated data table associated with the natural language text from a preset database based on the natural language text.

[0053] The task-associated field is determined from the preset database based on the natural language text representing the query task, and the fields associated with the natural language text are used to rewrite the natural language text to improve the accuracy of the final structured query statement generation.

[0054] The task-associated data table is determined from the preset database based on the natural language text representing the query task, and the data tables associated with the natural language text are used in the generation of the structured query statement.

[0055] In a specific embodiment, information understanding can be performed on the natural language text first, and then information recall and information sorting can be performed based on the information understanding. Specifically, the corresponding fields and data tables can be recalled from the preset database, and the fields and data tables can be sorted. According to the sorting result, further screening is performed to obtain the task-associated field and the task-associated data table.

[0056] The candidate fields and the candidate data tables have multiple recall methods, such as being recalled by entity similarity, being recalled by question similarity, and being recalled by question Elasticsearch retrieval. In specific scenarios, similarity retrieval and Elasticsearch retrieval can be performed according to key entities and questions for multi-path recall. Then, the similarity of each candidate field and each candidate data table to the natural language text is calculated, and the fields with a similarity greater than a preset similarity are determined as candidate fields and are recalled. The data tables with a similarity greater than a preset similarity are determined as candidate data tables and are recalled. In some other embodiments, the fields in the preset database can be sorted according to the similarity, such as being sorted from large to small, to obtain sorted fields, and the first M fields in the sorted fields are recalled as candidate fields. Similarly, the data tables in the preset database can also be sorted according to the similarity, such as being sorted from large to small, to obtain sorted data tables, and the first N data tables in the sorted data tables are recalled as candidate data tables. Finally, the task-related fields and the task-related data tables are determined according to the recalled candidate fields and the candidate data tables.

[0057] In some embodiments, information recall can be performed in a collaborative filtering manner. Collaborative filtering can include user-user collaborative filtering, item-item collaborative filtering, matrix decomposition, and the like. The user-user collaborative filtering (User-User Collaborative Filtering) specifically refers to finding similar users and recommending items according to user preferences. The item-item collaborative filtering (Item-Item Collaborative Filtering) specifically refers to making recommendations based on the similarity between items. The matrix decomposition (Matrix Factorization) such as singular value decomposition (SVD) and probability mass function (PMF) is used to discover the hidden features of users and items.

[0058] If user-user collaborative filtering is used, a user vector is first constructed according to the historical operation records (such as query, browsing, etc.) of the user on the natural language text, and then the similarity between different users is calculated to find a user group similar to the current user. Analyze the fields and data tables frequently queried by the similar user group in the preset database, and take these fields and data tables as the recall candidate objects. If item-item collaborative filtering is used, the similarity between each field and data table in the preset database is calculated. When the natural language text is related to a certain field or data table, other fields and data tables with high similarity are recalled. If matrix factorization (such as SVD (Singular Value Decomposition, Singular Value Decomposition)) is used, the interaction matrix of the user and the field, data table in the preset database is decomposed to obtain the latent feature vector of the user and the field, data table. Map the natural language text into this latent feature space to find the matching field and data table for recall.

[0059] In specific scenarios, the natural language question text can also be rewritten before information recall to enable the model to better understand the question. The rewriting here is the rewriting of general questions, which specifically refers to adjusting and processing the natural language question text, such as rewriting the natural language question text into a clearer text representation according to the context of the user's question; for example, the natural language text may contain errors, which can be corrected.

[0060] Among them, the first is to set a few-shot-based large model to directly rewrite the question; the second is to collect data for model training and rewrite through the trained model.

[0061] Optionally, in the embodiment, the step of "determining at least one task-associated field and at least one task-associated data table associated with the natural language text from the preset database based on the natural language text" can include: recalling at least one candidate field and at least one candidate data table from the preset database based on the natural language text; calculating the similarity of each candidate field and each candidate data table with the natural language text, respectively, to determine the task-associated field from the candidate field and the task-associated data table from the candidate data table according to the similarity.

[0062] Wherein, the candidate fields and candidate data tables can be recalled based on the natural language text. There are multiple recall methods for the candidate fields and candidate data tables, such as recalling through entity similarity, recalling through question similarity, and recalling through question Elasticsearch retrieval. In specific scenarios, multi-path recall can be performed according to key entities and questions through similarity retrieval and Elasticsearch retrieval.

[0063] In some embodiments, based on the natural language text, the process of recalling at least one candidate field and at least one candidate data table from the preset database using a hybrid recall strategy specifically includes: first, using a deep learning model to perform semantic understanding and feature extraction on the natural language text, and matching the extracted features with the features of the fields and data tables in the preset database. At the same time, in combination with a rule engine, specific recall rules are formulated according to the keywords and grammatical structures in the natural language text. For example, when the natural language text contains "time" related vocabulary, preferentially recall the data tables and fields related to time. During the recall process, a caching mechanism is introduced, and frequently recalled fields and data tables are cached to reduce the time overhead of repeated queries.

[0064] Optionally, in the present embodiment, the step of "recalling at least one candidate field and at least one candidate data table from the preset database based on the natural language text" can include: performing entity recognition processing on the natural language text to obtain an entity set corresponding to the natural language text; recalling at least one candidate field from the preset database based on the similarity between the entities in the entity set and the fields in the preset database; and recalling at least one candidate data table from the preset database based on the similarity between the entities in the entity set and the table information of the data tables in the preset database.

[0065] Wherein, the present embodiment can recall candidate fields and candidate data tables through entity similarity recall, which is a method of recalling candidate fields and candidate data tables from the preset database according to the entities contained in the natural language question text proposed by the user by calculating the similarity between the entities. The implementation methods include similarity calculation recall based on word vectors and similarity calculation recall based on knowledge graphs. The specific implementation methods of entity similarity recall include but are not limited to: similarity calculation recall based on word vectors, similarity calculation recall based on knowledge graphs, and determining similarity based on the vector distance between entity feature vectors and field and data table feature vectors.

[0066] Among them, entity recognition (Named Entity Recognition, NER) can include named entity recognition and entity extraction. Among them, named entity recognition is used to locate and classify named entities in text into predefined categories, such as personnel, organizations, locations, time expressions, quantities, percentages, etc.; specifically, the task of named entity recognition is to identify three categories (entity class, time class and number class) and seven small categories (person name, organization name, place name, time, date, currency and percentage) named entities in the text to be processed. In the embodiments of the present application, through named entity recognition, entities of preset entity types are obtained, such as person name and place name type entities, and then each entity can be obtained through entity extraction.

[0067] Among them, the specific entity recognition method includes but is not limited to jieba (Jieba) word segmentation, self-defined word library, large model prompt (prompt) word segmentation, etc.

[0068] Optionally, in the present embodiment, the step of "recalling at least one candidate field from the preset database based on the similarity between the entities in the entity set and the fields in the preset database" can include: performing feature extraction on the entities in the entity set to obtain feature vectors of the entities in the entity set; performing feature extraction on the fields in the preset database to obtain feature vectors of the fields in the preset database; determining the similarity between the entities and the fields based on the vector distance between the feature vectors of the entities and the feature vectors of the fields; and recalling at least one candidate field from the preset database according to the similarity.

[0069] The step of "recalling at least one candidate data table from the preset database based on the similarity between the entities in the entity set and the table information of the data tables in the preset database" can include: performing feature extraction on the table information of the data tables in the preset database to obtain feature vectors corresponding to the data tables in the preset database; determining the similarity between the entities and the data tables based on the vector distance between the feature vectors of the entities and the feature vectors of the data tables; and recalling at least one candidate data table from the preset database according to the similarity.

[0070] Among them, the present embodiment can recall candidate fields and candidate data tables based on the similarity calculation of word vectors, and the similarity calculation of word vectors is: using a pre-trained word vector model to convert entities into vectors and calculating the similarity between entities. Among them, the smaller the vector distance, the higher the similarity; on the contrary, the larger the vector distance, the lower the similarity.

[0071] The feature vector can be extracted by a neural network model, which can be of any type, such as a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), a Transformer, a Bidirectional Encoder Representations from Transformers (BERT), or the like, but it should be understood that the neural network of the present embodiment is not limited to the above-mentioned types. In other embodiments, feature extraction can also be performed by a decision tree, a random forest, a gradient boosting machine (GBM), or the like. A decision tree can infer a target value from data features by learning simple decision rules; a random forest integrates multiple decision trees to improve the accuracy and stability of the model; and a gradient boosting machine is to minimize a loss function by iteratively adding a weak prediction model.

[0072] The table information of the data table can be the table name of the data table, or a field in the data table, and the present embodiment is not limited in this regard.

[0073] In some embodiments, fields with a similarity greater than a preset similarity can be determined as candidate fields and recalled, and data tables with a similarity greater than a preset similarity can be determined as candidate data tables and recalled. In other embodiments, the fields in the preset database can be sorted according to the similarity, such as in descending order, to obtain sorted fields, and the first M fields in the sorted fields can be recalled as candidate fields. Similarly, the data tables in the preset database can also be sorted according to the similarity, such as in descending order, to obtain sorted data tables, and the first N data tables in the sorted data tables can be recalled as candidate data tables.

[0074] Optionally, in the present embodiment, the step of “recalling at least one candidate field from the preset database based on the similarity between the entities in the entity set and the fields in the preset database” can include: obtaining structural relationship information of the fields in the preset database and the entities in the entity set in the preset knowledge graph; determining semantic relevance between the fields and the entities based on semantic information of the fields and semantic information of the entities; calculating the similarity between the fields and the entities based on the structural relationship information and the semantic relevance; and recalling at least one candidate field from the preset database according to the similarity.

[0075] In the embodiment, the similarity between the field and the entity can be calculated based on a knowledge graph. The similarity calculation based on the knowledge graph is as follows: the similarity between the entities is calculated based on the relationship between the entities in the knowledge graph. The preset knowledge graph in the embodiment can be a graph model. The graph model includes nodes and edges between the nodes. The edges between the nodes represent the relationship between the two connected nodes. Specifically, the graph model can be a graph embedding model or a graph convolutional network (GCN). The graph embedding model can embed the graph nodes (users or items) into a low-dimensional vector space, such as Node2Vec, DeepWalk (random walk model), and the like. The graph convolutional network can be used to learn the representation of the graph structure data.

[0076] In the embodiment, the similarity between the field and the entity can be calculated based on a knowledge graph. The similarity calculation based on the knowledge graph is as follows: the similarity between the entities is calculated based on the relationship between the entities in the knowledge graph. The preset knowledge graph in the embodiment can be a graph model. The graph model includes nodes and edges between the nodes. The edges between the nodes represent the relationship between the two connected nodes. Specifically, the graph model can be a graph embedding model or a graph convolutional network (GCN). The graph embedding model can embed the graph nodes (users or items) into a low-dimensional vector space, such as Node2Vec, DeepWalk (random walk model), and the like. The graph convolutional network can be used to learn the representation of the graph structure data.

[0077] After obtaining the structural relationship information and the semantic correlation of the field and the entity, the structural relationship information and the semantic correlation can be fused to obtain the similarity between the field and the entity. The fusion method can be weighted summation or the like.

[0078] In some embodiments, the fields with a similarity greater than a preset similarity can be determined as candidate fields and recalled. In other embodiments, the fields can also be sorted based on the similarity, and the candidate fields can be recalled based on the sorting result.

[0079] Optionally, in the embodiment, the step of “recalling at least one candidate data table from the preset database based on the similarity between the entities in the entity set and the table information of the data tables in the preset database” can include: obtaining structural relationship information of the table information of the data tables in the preset database and the entities in the entity set in the preset knowledge graph; determining semantic correlation between the data tables and the entities based on semantic information of the table information of the data tables and semantic information of the entities; calculating the similarity between the data tables and the entities based on the structural relationship information and the semantic correlation; and recalling at least one candidate data table from the preset database based on the similarity.

[0080] In the embodiment, the candidate data table can be recalled based on the similarity calculation of the knowledge graph. The structural relationship information of the table information of the data tables and the entities in the preset knowledge graph can include the number of connection paths of the corresponding nodes in the preset knowledge graph.

[0081] After obtaining the structural relationship information and semantic correlation between the data table and the entity, the structural relationship information and the semantic correlation can be fused to obtain the similarity between the data table and the entity. The fusion manner can be various, such as weighted summation.

[0082] In some embodiments, the data table with a similarity greater than a preset similarity can be determined as a candidate data table, and the candidate data table can be recalled. In other embodiments, the candidate data tables can also be sorted based on the similarity, and the candidate data tables can be recalled according to the sorting result.

[0083] Optionally, in the embodiment, the step of “recalling at least one candidate field and at least one candidate data table from the preset database based on the natural language text” can include: calculating a matching degree between the natural language text and each preset question, each preset question being associated with at least one data table in the preset database; determining a similar question from the preset questions according to the matching degree; determining the data table associated with the similar question as a candidate data table; and recalling at least one candidate field from the preset database based on the natural language text.

[0084] In the embodiment, the candidate data table can be recalled by question similarity. Specifically, the preset database can further include preset questions, each of which is associated with a data table. The embodiment can recall the data table by calculating the similarity between the natural language question text proposed by the user and each preset question.

[0085] The question similarity recall is a method of recalling the data table related to the natural language question text by calculating the similarity between the natural language question text proposed by the user and each preset question in the preset database. The implementation manner includes word vector-based similarity calculation recall, text similarity algorithm-based calculation recall, etc.

[0086] The implementation manner of the question similarity recall can include word vector-based similarity calculation recall and text similarity algorithm-based calculation recall. The word vector-based similarity calculation recall specifically uses a pre-trained word vector model to convert the question into a vector and calculates the cosine similarity between the questions, so as to recall the candidate data table according to the cosine similarity between the questions. The text similarity algorithm-based calculation can include calculating the similarity between the questions by TF-IDF, BM25, etc., and then recalling the candidate data table according to the similarity. BM25 is an algorithm used to evaluate the relevance between search words and documents.

[0087] The preset question with a matching degree greater than a preset matching degree can be determined as a similar question. The preset questions can also be sorted according to the matching degree, such as sorting from large to small, obtaining the sorted questions, and then determining the first K preset questions in the sorted questions as similar questions.

[0088] Optionally, in the embodiment, the step of "recalling at least one candidate field and at least one candidate data table from the preset database based on the natural language text" can include: performing data analysis on the data tables in the preset database to determine the key information of the data tables; generating a retrieval index of the data tables according to the key information of the data tables; creating a retrieval index library based on the retrieval index, the retrieval index library including a mapping relationship between the retrieval index and the data tables; and performing data table and field searching in the retrieval index library based on the natural language text to obtain at least one candidate data table and candidate field.

[0089] In the embodiment, the candidate data table and the candidate field can be recalled by question Elasticsearch retrieval. Specifically, the question Elasticsearch retrieval refers to taking the natural language question text proposed by the user as a query statement, performing retrieval through an Elasticsearch search engine, and returning relevant tables and fields.

[0090] The question Elasticsearch retrieval is a method of taking the natural language question text proposed by the user as a query statement, performing retrieval through an Elasticsearch search engine, and returning relevant tables and fields. The natural language question text needs to be processed by word segmentation, removal of stop words, etc. before searching for data tables and fields in the retrieval index library.

[0091] In the embodiment, the key information of the data tables can be used as the retrieval index of the data tables, and then the retrieval index library can be created according to the retrieval index, and one retrieval index can correspond to one or more data tables. Specifically, the natural language question text can be processed by word segmentation, removal of stop words, etc. before being used as a query statement for retrieval.

[0092] In the embodiment, Elasticsearch is a distributed search and analysis engine. In the embodiment, inverted indexes can be established by Elasticsearch, and Term Frequency-Inverse Document Frequency (TF-IDF) functions can be used to analyze the data tables in the preset database to obtain the key information of the data tables. For the TF-IDF, the term frequency is the frequency of each word appearing in the text, and the inverse document frequency is used to reflect the importance of a word. The inverse document frequency can reflect the frequency of a word appearing in all texts. If a word appears in many texts, the value of the inverse document frequency of the word is low; on the contrary, if a word appears only once or twice in all texts, the value of the inverse document frequency of the word is high.

[0093] Specifically, the importance of the keywords in the document can be evaluated by using a TF-IDF (Term Frequency-Inverse Document Frequency) model to determine the key information; and a latent Dirichlet allocation (LDA) can be used to discover hidden topics in the document set to determine the key information.

[0094] Optionally, in some embodiments, the step of "calculating the similarity of each candidate field and each candidate data table with the natural language text respectively, and determining the task-related field from the candidate fields and the task-related data table from the candidate data tables according to the similarity" can include: using a deep learning-based similarity calculation model to calculate the similarity of each candidate field and each candidate data table with the natural language text. The model is based on BERT, encodes the natural language text, candidate fields and candidate data tables, and then calculates the similarity scores between them through a fully connected layer. In the training process, a large amount of labeled data is used to fine-tune the model and optimize the parameters of the model. At the same time, a dynamic weight adjustment mechanism is introduced to dynamically adjust the weights of various features in similarity calculation according to different business scenarios and data characteristics. For example, in some business scenarios, the matching degree of the field name with the natural language text has a higher weight; while in other scenarios, the matching degree of the data type of the field with the query requirement has a higher weight. According to the calculated similarity scores, the threshold screening and sorting method is used to determine the task-related field from the candidate fields and the task-related data table from the candidate data tables.

[0095] Optionally, in the present embodiment, the step of "calculating the similarity of each candidate field and each candidate data table with the natural language text respectively, and determining the task-related field from the candidate fields and the task-related data table from the candidate data tables according to the similarity" can include: filtering the candidate data tables according to a preset data strategy to obtain filtered candidate data tables; detecting the relevance of each candidate field and each filtered candidate data table with the natural language text through a first preset model to determine the task-related field from each candidate field and the associated data table from each filtered candidate data table; fusing the table identifier information and the table field of the associated data table to obtain table information; fusing the table information with the entity of the natural language text through a second preset model to obtain fused information; extracting the feature information of the fused information to predict the similarity of the associated data table with the natural language text based on the feature information; and determining the task-related data table from the associated data table according to the similarity.

[0096] The preset data strategy can be set according to actual business requirements, for example, the preset data strategy can be some field rules. Specifically, the candidate data table includes related data tables of App (Application) and Pc (Personal Computer), and the preset data strategy can be to filter out the data tables related to the Pc.

[0097] The table identifier information of the associated data table can be the table name of the associated data table. Specifically, there are various ways to fuse the table identifier information and the table field of the associated data table, for example, the table name and the table field of the associated data table can be spliced to obtain the table information.

[0098] There are various ways to fuse the table information and the entity of the natural language text, for example, the table information and the entity of the natural language text can be spliced to obtain fused information, and then the second preset model is used to predict the similarity between the associated data table and the natural language text based on the fused information.

[0099] In some embodiments, the associated data table with a similarity greater than a preset similarity can be determined as the task-associated data table. In other embodiments, the associated data tables can be sorted according to the similarity, for example, sorted from large to small, to obtain sorted associated data tables, and the first Q associated data tables in the sorted associated data tables are determined as the task-associated data tables.

[0100] In specific scenarios, the first preset model can be an Encoder model, and the second preset model can be a cross-encoder model. In this embodiment, the Encoder model can be used to determine whether the field and the data table are related to the natural language question text by using the 01 classification method. If a candidate field is determined to be related, the candidate field is determined as the task-associated field. If a candidate data table is determined to be related, the candidate data table is determined as the associated data table. Then, the cross-encoder model is used to fuse the table information and the entity of the associated data table to determine the similarity between the table information and the entity information, so as to determine the task-associated data table.

[0101] It can be understood that the first preset model and the second preset model can also be other types of neural network models, and this embodiment does not limit this.

[0102] 103. Rewriting the natural language text according to the task-associated field to obtain a rewritten natural language text.

[0103] The rewritten natural language text is obtained by rewriting the original natural language text according to the task association field, so that the natural language text is easier to be understood by the model, thereby improving the accuracy of the final SQL generation.

[0104] The embodiment can rewrite the natural language text so that the natural language text can be better understood by the model. Specifically, the entity in the natural language text can be rewritten based on the task association field by using a question rewriting model, so as to rewrite the natural language text into a style that is easier for the model to understand, thereby improving the accuracy of the final SQL generation.

[0105] In some embodiments, the natural language text can be rewritten according to the task association field by using a question rewriting model based on reinforcement learning to obtain the rewritten natural language text. The training data of the model includes a large amount of natural language text, corresponding task association fields and artificially rewritten texts. In the training process, a policy gradient algorithm is used, and the similarity between the rewritten text and the target text is used as a reward signal to continuously optimize the rewriting strategy of the model. At the same time, an adversarial training mechanism is introduced to train a discriminator to distinguish whether the rewritten text is artificially rewritten or generated by the model, so as to encourage the generator to generate a text closer to the artificially rewritten text. For example, in the training process, the discriminator judges the difference between the rewritten text generated by the model and the artificially rewritten text, and the generator adjusts its parameters according to the feedback of the discriminator to improve the quality of the rewriting. The question rewriting model can be a neural network model, such as an LLM model (Large Language Model).

[0106] The question rewriting model can be a neural network model, such as an LLM model (Large Language Model). Through the question rewriting model, the entity corresponding to the task association field in the natural language text can be identified, and then the identified entity can be rewritten based on the task association field, thereby obtaining the rewritten rewritten natural language text.

[0107] In a specific embodiment, the natural language question text is "How many times did YY province users watch cid_title like 'TV drama XX' in the past 7 days on a certain video app?", the task association fields are respectively: TV drama XX: A.cid_title (dimension value), YY province: B.province (dimension value), and play times: A.vfinish_cnt (field value). The rewritten natural language text obtained by question rewriting is: "How many times did YY province users watch cid_title like 'TV drama XX' in the past 7 days on a certain video app?"; or the rewritten natural language text obtained by question rewriting can be: "How many times did YY province users watch cid_title like 'TV drama XX' in the past 7 days on a certain video app?"

[0108] wherein A.cid_title represents the field cid_title (title) in the data table A; B.province represents the field province (province) in the data table B; and A.vfinish_cnt represents the field vfinish_cnt (play times) in the data table A.

[0109] In specific embodiments, for the natural language question text, the question rewriting can be performed by dimension hierarchy recall + large model + fewshot, wherein the dimension hierarchy recall specifically recalls the entity to be rewritten, so that the large model + fewshot performs question rewriting based on the entity to be rewritten. Prompt example 1 is as follows:

[0110] Question: How many times did YY province users watch cid_title like 'TV drama XX' in the past 7 days on a certain video app?

[0111] Entity corresponding field information: TV drama XX: A.cid_title (dimension value); YY province: B.province (dimension value); and play times: A.vfinish_cnt (field value)

[0112] Output: How many times did YY province users watch cid_title like 'TV drama XX' in the past 7 days on a certain video app?

[0113] Example 2:

[0114] Question: Which province has the most users watching cid_title like 'Animation XX' on a certain video app?

[0115] Entity corresponding field information: province: B.province (dimension value); Animation XX: A.cid_title (dimension value); and play times: A.vfinish_cnt (field value)

[0116] Output: The province with the highest vfinish_cnt in the cid_title like 'Animation XX' in a certain video App

[0117] Example 3:

[0118] Question: The trend of the proportion of the number of plays of the drama category in the total number of plays in a certain video App in the past week

[0119] Entity corresponding field information: Drama category: C.el_type_name (dimension value); Animation XX: A.cid_title (dimension value); Number of plays: A.vfinish_cnt (field value)

[0120] Output: The trend of the proportion of the vfinish_cnt of el_type_name = 'drama' in a certain video App in the past week

[0121] Example 4:

[0122] Question: Comparison of the number of plays of 'Drama 1' and 'Drama 2' in a certain video App in the past week

[0123] Entity corresponding field information: Drama 1: A.cid_title (dimension value); Drama 2: A.cid_title (dimension value); Number of plays: A.vfinish_cnt (field value)

[0124] Output: Comparison of the vfinish_cnt of cid_title like 'Drama 1' and cid_title like 'Drama 2' in a certain video App in the past week

[0125] Example 5:

[0126] Question: The number of people who watched 'Drama 1' and 'Drama 2' in a certain video App

[0127] Entity corresponding field information: Drama 1: A.cid_title (dimension value); Drama 2: A.cid_title (dimension value); Number of plays: A.qimei36 (field value)

[0128] Output: The number of unique qimei36 in a certain video App that watched cid_title like 'Drama 1' and cid_title like 'Drama 2' at the same time.

[0129] Optionally, the LLM model fine-tuning training can also be performed by collecting the rewriting data, and the question rewriting is performed by the trained model, and the data pattern of this scheme is as follows:

[0130] Input: the number of times of playing the video App in the past 7 days by the YY province users watching the TV series XX;

[0131] table_info: XXX

[0132] link_info: TV series XX: A.cid_title (dimension value); YY province: B.province (dimension value); number of times of playing: A.vfinish_cnt (field value)

[0133] Output: the number of times of playing the video App in the past 7 days by the YY province users watching the TV series XX.

[0134] Optionally, in the embodiment, before the step of "rewriting the natural language text according to the task association field to obtain the rewritten natural language text", it can also include: filtering the task association data table and the task association field based on the natural language text by the information optimization model to obtain the filtered task association data table and the filtered task association field; detecting whether the filtered task association field can perform the structured query task of the natural language text based on the filtered task association data table; filtering the filtered task association field based on the detection result to obtain the filtered task association field.

[0135] Among them, the embodiment can further perform information simplification processing on the data table and the field by the information optimization model after recalling and sorting the data table and the field, so as to delete irrelevant information, select the most relevant table and field information of the natural language question text proposed by the user, and reduce the interference of irrelevant information on the query sentence generation model.

[0136] The information optimization model is used to further simplify the data table and field information, delete irrelevant information, select the most relevant table and field information of the natural language question text proposed by the user, and reduce the interference on the query sentence generation model after recalling and sorting the data table and field. The model can be a cross_encoder model or an LLM model. Specifically, the matching scores of the natural language text with each task association data table and each task association field can be calculated by the information optimization model, and then the task association data table and the task association field are further filtered according to the matching scores.

[0137] Among them, the training scheme of the cross_encoder model is simple, and the data is easy to collect. The LLM model is biased towards fine selection.

[0138] The embodiment can also perform a pre-check operation on the screened task association field. Specifically, it is detected whether the screened task association field can perform the structured query task of natural language text, that is, it is determined whether the table field information can complete the corresponding SQL task. If it cannot be completed, the table field information is rejected in advance. Specifically, if it is detected that a certain task association field cannot complete the corresponding SQL task, the task association field is filtered out. If it is detected that a certain task association field can complete the corresponding SQL task, the task association field is retained.

[0139] Specifically, if the previous step adopts a cross-encoder model scheme, a score threshold strategy can be used to determine whether the field can solve the user's problem. If the previous step adopts a large model + prompt or model fine-tuning, a large model + prompt can be used to determine whether the SQL task can be completed.

[0140] In a specific embodiment, after information recall sorting, there is still a large amount of irrelevant information, which needs to be refined by an information optimization model. A batch of training data can be prepared to use the LLM fine-tuning method to refine.

[0141] In some embodiments, a large model + prompt can also be used to solve the information accuracy problem. Specifically, the table information obtained in the previous step can be collected, the entity information can be assembled, and the prompt can be used to let the large model solve the information accuracy problem. The prompt can be as follows:

[0142] Prompt skeleton example:

[0143] {table_info}

[0144] Entity information: {entities}

[0145] ###Requirements

[0146] You are a natural language processing expert and a data analyst in the Internet industry. Please match the entity with the table field needed in data analysis.

[0147] Note that an entity may require multiple fields, and the fields should be as comprehensive as possible, and as many fields as possible within a single table should be selected.

[0148] Note that all table names must be the original names, and the output of the field name must be the original field name in the table.

[0149] ###Please comply with the following specifications

[0150] 1. The number of elements in the json is {numbers}, corresponding to {limitation}, {metric};

[0151] 2. yyy1 in the json is the table field that the computing entity needs to select, which requires matching one or more fields that can solve the problem of the entity; must be answered in the form of table name.field name, and the table name cannot be omitted!

[0152] 3. Please summarize your output only in the given json format: {output_format}

[0153] 104. Obtain prompt information for the rewritten natural language text, and the prompt information is used to guide the query statement generation model to generate a structured query statement.

[0154] Among them, the prompt information is information used to guide the query statement generation model to generate a structured query statement, which can be generated based on the rewritten natural language text through a neural network model, or can be obtained by selecting from a preset prompt library.

[0155] The preset prompt library is a library containing a large number of preset prompt information, and different preset prompt libraries contain different types of prompt information, such as a preset time condition library, a preset join table condition library, a preset calculation logic library, and a preset few-shot library. The target prompt library can be selected according to the keywords, semantics, domain, and other information of the rewritten natural language text, and then the prompt information is determined.

[0156] The query statement generation model is a neural network model, which is not limited in type, and is used to convert natural language problems into structured query statements (SQL). It generates a structured query statement corresponding to the rewritten natural language text based on task-related fields, task-related data tables, prompt information, and problem type information.

[0157] The query statement generation model can be a neural network model, which is not limited in type, such as a convolutional neural network (CNN), a residual network (ResNet), and a densely connected convolutional network (DenseNet). However, it should be understood that the neural network of the present embodiment is not limited to the above-mentioned several types. The query statement generation model can be used to generate SQL query statements.

[0158] In some embodiments, the prompt information can be generated by a neural network model based on the rewritten natural language text. In other embodiments, the prompt information can also be selected from a preset prompt library based on the rewritten natural language text. The preset prompt library can include a large number of preset prompt information.

[0159] Optionally, in the present embodiment, the step of "obtaining prompt information for the rewritten natural language text" can include: obtaining a query statement generation model; selecting a target prompt library from each preset prompt library according to the rewritten natural language text, and determining the prompt information of the rewritten natural language text according to the correspondence between the information in the target prompt library and the rewritten natural language text.

[0160] In the present embodiment, the different preset prompt libraries can include different types of prompt information. The corresponding target prompt library can be selected from each preset prompt library according to the keywords, semantics, domain, and other information contained in the rewritten natural language text. Then, appropriate prompt information can be retrieved from the target prompt library. In order to improve the retrieval efficiency, efficient retrieval algorithms and data structures such as inverted index can be used.

[0161] During the retrieval process, the system will calculate the matching degree between the user's question and the information in the retrieval library (i.e., the target prompt library), and filter out the prompt information most relevant to the user's question. The matching degree calculation can use various methods, such as cosine similarity, Jaccard similarity, etc.

[0162] In some embodiments, the target prompt library selection method based on knowledge graph can be used according to the rewritten natural language text. First, the entities and relationships in the rewritten natural language text are mapped to the knowledge graph. Then, according to the semantic association and reasoning rules in the knowledge graph, the target prompt library most relevant to the rewritten natural language text is determined. For example, when the rewritten natural language text contains "sales data" related entities, the prompt library related to "sales data" such as the preset sales calculation logic library can be found through the knowledge graph. At the same time, the user's historical query records and preference information are used to further filter and sort the selected target prompt library, improving the accuracy of selection.

[0163] Optionally, in this embodiment, the preset prompt library includes a preset time condition library, a preset join table condition library, a preset calculation logic library, and a preset few-shot library; the step of "selecting a target prompt library from each preset prompt library according to the rewritten natural language text, and determining the prompt information of the rewritten natural language text according to the correspondence between the information in the target prompt library and the rewritten natural language text" can include: when the rewritten natural language text contains a time entity, selecting the preset time condition library as the target prompt library to retrieve time partition condition information corresponding to the time entity from the target prompt library, and generating prompt information according to the time partition condition information; when the rewritten natural language text is related to a data table connection, selecting the preset join table condition library as the target prompt library to retrieve join table condition information corresponding to the task-associated data table from the target prompt library, and generating prompt information based on the join table condition information; when the rewritten natural language text contains a calculation index, selecting the preset calculation logic library as the target prompt library to retrieve calculation logic information corresponding to the calculation index from the target prompt library, and generating prompt information according to the calculation logic information; when the rewritten natural language text is related to a few-shot learning problem, selecting the preset few-shot library as the target prompt library to retrieve a structured query generation strategy corresponding to the few-shot learning problem from the target prompt library, and generating prompt information based on the structured query generation strategy.

[0164] The preset time condition library can include various time partition condition information, such as by year, by quarter, by month, etc. When a user proposes a natural language question text related to time partition, the system can detect that the natural language question text matches the preset time condition library, then retrieve corresponding time partition condition information from the preset time condition library, and provide a pre-prompt for the user. Specifically, in this embodiment, the time partition condition information can be directly used as prompt information, or the time partition condition information can be assembled with a preset prompt format template to obtain prompt information.

[0165] The preset join table condition library can include various connection conditions between databases, such as inner join, outer join, cross join, etc. When a user proposes a natural language question text related to a data table connection, the system can detect that the natural language question text matches the preset join table condition library, then retrieve corresponding join table condition information from the preset join table condition library, and provide a pre-prompt for the user. Specifically, in this embodiment, the join table condition information can be directly used as prompt information, or the join table condition information can be assembled with a preset prompt format template to obtain prompt information.

[0166] The pre-designed calculation logic library can include various commonly used index calculation methods, such as mean, median, mode, etc. When the user proposes a natural language question text related to index calculation, the system can detect that the natural language question text is related to index calculation, and then retrieve the corresponding calculation method from the pre-designed calculation logic library and provide a pre-prompt for the user. Specifically, in this embodiment, the retrieved calculation logic information can be directly used as the prompt information, or the calculation logic information can be assembled with a pre-set prompt format template to obtain the prompt information.

[0167] The pre-set few-shot sample library can include various methods and techniques for few-shot learning. When the user proposes a natural language question text related to few-shot learning, the system can detect that the natural language question text is related to few-shot learning, and then retrieve the corresponding few-shot learning method and technique (specifically, a structured query generation strategy) from the pre-set few-shot sample library and provide a pre-prompt for the user. Specifically, in this embodiment, the retrieved structured query generation strategy can be directly used as the prompt information, or the structured query generation strategy can be assembled with a pre-set prompt format template to obtain the prompt information.

[0168] 105. classifying the rewritten natural language text to obtain question type information of the rewritten natural language text.

[0169] The question type information is information obtained by classifying the rewritten natural language text, and can be classified according to question asking method paradigms and SQL difficulty of the natural language question text, and can help the query statement generation model to better generate a structured query statement. The asking method paradigm is a basis for classifying the asking method of the natural language question text, and the asking method paradigm of the rewritten natural language text can be determined through an asking method paradigm classification model. The classification basis is usually to induce special asking method categories that are difficult for large models to correctly solve, such as maximum value, proportion, same period, ring ratio, and others.

[0170] Specifically, in this embodiment, the natural language question text can be classified according to the question asking method paradigm and the SQL difficulty of the question, to obtain corresponding question type information, which can help the query statement generation model to better generate a SQL query statement. It can be understood that other classification bases can also be used for classification, which are not limited in the present application.

[0171] Optionally, in this embodiment, the step of "classifying the rewritten natural language text into a question type to obtain question type information of the rewritten natural language text" can include: determining the question paradigm corresponding to the rewritten natural language text; determining the generation difficulty level of the query statement corresponding to the task-related data table according to the data structure association information between the task-related data tables; and determining the question type information of the rewritten natural language text from each preset question type information based on the question paradigm and the generation difficulty level, wherein each preset question type information corresponds to a preset question paradigm and a preset generation difficulty level.

[0172] The generation difficulty level is the generation difficulty level of the query statement corresponding to the task-related data table determined according to the data structure association information (such as nesting and table association information) between the task-related data tables, and can be divided into simple questions (without table association and nesting), nested questions (only requiring nested subqueries), table association questions (only requiring table association), and table association and nesting questions, and the higher the level, the greater the generation difficulty.

[0173] The question paradigm of the rewritten natural language text can be determined through a question paradigm classification model, which can be a neural network model without limitation on its type. In this embodiment, the classification of the question paradigm can be based on special question categories that are difficult for an inductive large model to correctly solve. Specifically, the question paradigm classification can include maximum value, proportion, same period, ring ratio, and other categories. The question paradigm classification can be performed through a large model and a prompt, and a classification model can be trained using question classification data, and the question paradigm classification can be performed through the trained model.

[0174] In some embodiments, the question paradigm corresponding to the rewritten natural language text can be determined using a question paradigm classification model based on multi-modal feature fusion. This model fuses the semantic features, syntactic features, and contextual features of the text, and extracts and processes these features through a convolutional neural network (CNN) and a recurrent neural network (RNN). During training, a large-scale labeled data set is used for training, and transfer learning and fine-tuning techniques are used to initialize model parameters using a pre-trained language model to improve the training efficiency and classification accuracy of the model. At the same time, a model evaluation and optimization mechanism is introduced to evaluate and adjust the model regularly, and the structure and parameters of the model are optimized according to the evaluation results.

[0175] The data structure association information can include nesting and table association information, etc. Specifically, the generation difficulty level of the query statement can be determined according to whether table association and nesting are required. In specific scenarios, SQL difficulty classification can be performed through a large model and a prompt, and a classification model can be trained using question classification data, and SQL difficulty classification can be performed through the trained model.

[0176] In some embodiments, a Graph Neural Network (GNN) can be used to model the structure of task-related data tables based on the data structure relationships between them, thereby more accurately determining the generation difficulty level of the query statements corresponding to each task-related data table. The GNN can capture the complex relationships and dependencies between data tables, and calculate the complexity score of each data table through the propagation and aggregation of node and edge features. Simultaneously, query statement complexity indicators, such as the number of query conditions, the number of nesting levels, and the use of aggregate functions, are considered to comprehensively evaluate the generation difficulty level of the query statements. For example, when a query statement contains multiple levels of nesting and multiple aggregate functions, the generation difficulty level increases accordingly.

[0177] Optionally, in this embodiment, the step "determine the generation difficulty level of the query statement corresponding to the task-related data table based on the data structure association information between the task-related data tables" may include: performing nesting and join detection on the task-related data tables; and determining the generation difficulty level of the query statement corresponding to the task-related data table based on the detection results.

[0178] Specifically, if the detection result indicates that neither table joins nor nesting are required, the target natural language question text can be determined as a simple question, thus classifying the query generation difficulty level as Level 1. If the detection result indicates that only table joins or only nesting are required, the target natural language question text can be determined as a medium-difficulty question, thus classifying the query generation difficulty level as Level 2. If the detection result indicates that both table joins and nesting are required, the target natural language question text can be determined as a complex question, thus classifying the query generation difficulty level as Level 3. The higher the level, the greater the difficulty in generating the query.

[0179] 106. Generate a model through query statements, based on task-related fields, task-related data tables, prompt information, and question type information, to generate a structured query statement for the rewritten natural language text.

[0180] The query statement generation model is used to convert natural language questions into SQL query statements. The rewritten natural language text, task-related fields, task-related data tables, prompts, and question type information are input into the query statement generation model. The model then outputs a structured query statement based on the rewritten natural language text. This structured query statement is used to perform a query in a pre-defined database to obtain the corresponding query results.

[0181] In some embodiments, the structured query statement of the rewritten natural language text can be generated by a query statement generation model, using a method combining attention mechanism and reinforcement learning, based on the task-related fields, the task-related data tables, the prompt information, and the question type information. The attention mechanism can make the model pay more attention to important information, improving the accuracy of the generated statement. Reinforcement learning can provide feedback and optimization to the model according to the execution results of the generated statement, prompting the model to generate more suitable query statements. For example, when the execution result of the generated query statement is inaccurate, the parameters of the model are adjusted through the reinforcement learning algorithm to improve the accuracy of the next generation.

[0182] Optionally, after the step of "generating, by a query statement generation model, a structured query statement of the rewritten natural language text based on the task-related fields, the task-related data tables, the prompt information, and the question type information", the structured query statement generation method can further include: checking the structured query statement; and when the checking result is passed, querying in a preset database based on the structured query statement to obtain a query result corresponding to the natural language text.

[0183] The checking of the structured query statement is an inspection operation performed on the structured query statement of the rewritten natural language text generated by the query statement generation model, and the purpose is to improve the accuracy of data query using SQL query statements. The checking methods include multi-level checking strategies such as syntax checking, semantic checking, and execution checking, and a machine learning model can be introduced to evaluate the checking result.

[0184] After the query statement generation model generates the structured query statement of the rewritten natural language text, the structured query statement of the rewritten natural language text can be further checked, which can improve the accuracy of data query using SQL query statements.

[0185] The checking method of the SQL query statement can be various, such as setting fields for self-checking, execution checking, etc.

[0186] The structured query statement can be checked by using a multi-level checking strategy. First, syntax checking is performed, the structured query statement is analyzed line by line using an SQL syntax parser, and it is checked whether the statement conforms to the SQL syntax specification, such as whether the use of keywords is correct, whether parentheses are matched, and the like. Then, semantic checking is performed, the query statement is compared with the metadata of the preset database, the table structure and field information of the database are traversed, and it is checked whether the tables and fields used in the query statement exist in the database. Finally, execution checking is performed, the generated query statement is put into an execution engine for execution, and error information in the execution process is recorded. If an error occurs in the execution process, possible execution error information is pulled for classification, such as non-existent fields, non-existent tables, and data type mismatches. At the same time, a machine learning model is introduced to predict and evaluate the checking result. A large number of historical query statements and their checking results are collected as training data to train a classification model (such as a decision tree classifier). When checking a new structured query statement, the features of the statement (such as the number of query conditions, the functions used, and the like) are input into the trained model, the model predicts the possible problem types of the statement, and potential problems are found in advance.

[0187] Optionally, in the embodiment, the step of "checking the structured query statement" can include: performing field analysis on the structured query statement to obtain a target field, and detecting whether the target field exists in a data table of a preset database; or executing the structured query statement through an execution engine; or detecting whether the structured query statement can process natural language text.

[0188] Specifically, in the embodiment, the structured query statement can be checked by self-checking. The self-checking is specifically a preliminary field checking of SQL to determine whether the data tables and fields used by the SQL are within the range of data tables of the preset database, thereby avoiding the problem of fabricating data tables and fields. Specifically, some SQL syntax analysis tools can be used to analyze the key table and field information.

[0189] In the embodiment, the structured query statement can also be checked by an execution engine. Specifically, the execution engine can be used to perform execution checking of the SQL, the generated SQL is put into the execution engine for execution, when an error occurs in the execution process, possible execution error information is pulled for correction, the specific execution engine error information is classified, and then specific error correction is performed.

[0190] Wherein, the embodiment can also detect whether the SQL query statement can solve the natural language problem text proposed by the user. The checking method is pos-check checking. Specifically, a large model can be used to determine whether the given problem and SQL can solve the user's problem. If it is determined that the user's problem cannot be solved, feedback is given to the user, and certain modification suggestions are given.

[0191] Optionally, after the step of "executing the structured query statement by the execution engine", the structured query statement generation method can further include:

[0192] When an error occurs in the execution process, error correction is performed on the structured query statement based on the error information. In some embodiments, when an error occurs in the execution process, error correction can be performed on the structured query statement based on the error information using a method combining rules and machine learning. First, according to a pre-set error rule library, the error information is classified and matched to find possible error causes and correction methods. For example, when the error information prompts "field does not exist", the possible alternative fields are found according to the rule library. At the same time, a machine learning model is used to analyze and predict the error information, and learn the characteristics and correction strategies of different types of errors. For example, a decision tree model is used to classify the error information, and the appropriate correction method is selected according to the classification result.

[0193] Wherein, the error correction can be performed by a neural network model, which can be of any type and can be obtained by training a large number of samples.

[0194] Optionally, after the step of "detecting whether the structured query statement can process the natural language text", the structured query statement generation method can further include: if it is detected that the structured query statement cannot process the natural language text, outputting feedback.

[0195] Wherein, the feedback can be output by a neural network model, which can be of any type and can be obtained by training a large number of related samples.

[0196] Optionally, in this embodiment, the query statement generation model comprises a basic module, an adaptive module, and an integration module, the adaptive module comprises sub-modules corresponding to respective preset question type information; the step of "generating, by the query statement generation model, the structured query statement of the rewritten natural language text based on the task association field, the task association data table, the prompt information, and the question type information" can comprise: obtaining feature interaction information by the basic module performing feature interaction on the rewritten natural language text, the task association field, the task association data table, the prompt information, and the question type information; for each sub-module, obtaining sub-feature interaction information output by the sub-module by performing feature interaction on the rewritten natural language text, the task association field, the task association data table, the prompt information, and the question type information; and obtaining the structured query statement of the rewritten natural language text by the integration module performing fusion processing on the feature interaction information and the sub-feature interaction information output by the respective sub-modules.

[0197] The adaptive module is part of the query statement generation model and comprises sub-modules corresponding to respective preset question type information, each sub-module can perform feature interaction on the rewritten natural language text, the task association field, the task association data table, the prompt information, and the question type information to obtain sub-feature interaction information, thereby assisting in generating the structured query statement, and the sub-module can be a LORA (Low-Rank Adaptation) module or the like. One LORA module is one expert model.

[0198] The integration module can perform fusion processing on the feature interaction information and the sub-feature interaction information output by the respective sub-modules in various ways, which are not limited in this embodiment. For example, the fusion processing can be weighted summation or the like.

[0199] The integration module is part of the query statement generation model and is used to fuse the feature interaction information output by the basic module and the sub-feature interaction information output by the respective sub-modules of the adaptive module to obtain the structured query statement of the rewritten natural language text, and specifically can be a MOE (Mixture of Experts) module, which improves the overall performance by weighted combination of outputs of multiple expert models.

[0200] LoRA (Low-Rank Adaptation of Large Language Models) is a low-parameter fine-tuning algorithm for large language models. Assuming that LoRA is adapted to downstream tasks, the full connection layer of the large model has a low intrinsic rank, i.e., contains a large amount of redundant information. A trainable rank decomposition matrix is injected into the full connection layer of the Transformer architecture, and the weights of the original pre-trained model are frozen, thereby greatly reducing the number of parameters involved in training.

[0201] Specifically, each sub-module includes a dimension reduction matrix A and a dimension increase matrix B; the step of "for each sub-module, performing feature interaction on the rewritten natural language text, the task association field, the task association data table, the prompt information and the question type information by the sub-module to obtain sub-feature interaction information output by the sub-module" can include: for each sub-module, performing splicing processing on the rewritten natural language text, the task association field, the task association data table, the prompt information and the question type information by the sub-module to obtain spliced information; performing dimension reduction processing on the spliced information by the dimension reduction matrix to obtain dimension-reduced information; performing dimension increase processing on the dimension-reduced information by the dimension increase matrix to obtain the sub-feature interaction information output by the sub-module.

[0202] During training, the parameters W of the pre-trained model are frozen and do not receive gradient updates, while the parameter matrices A and B introduced by LoRA contain trainable parameters and are parallel to the parameters W. Here, W and AW = BA are both multiplied by the same input, and the output vectors of each of them are summed by coordinates.

[0203] LoRA specifically increases a bypass next to the original model (specifically, a PLM, a pre-trained language model), performs a dimension reduction and then a dimension increase operation, as shown in FIG. 1c. During training, the parameters of the PLM are fixed, and only the dimension reduction matrix A and the dimension increase matrix B are trained, while the input and output dimensions of the model remain unchanged. When outputting, BA and the parameters of the PLM are superimposed, A is initialized with a random Gaussian distribution, that is, A = N(0, σ 2 ), and B is initialized with a 0 matrix, so that the bypass matrix is still a 0 matrix at the beginning of training.

[0204] Specifically, assuming that the parameter matrix of the pre-trained model is After adding the bypass, its update can be represented as formula (1):

[0205] wherein the rank r << min(d, k).

[0206] Optionally, before the step of "performing feature interaction on the rewritten natural language text, the task-related field, the task-related data table, the prompt information and the question type information by the basic module to obtain feature interaction information", the structured query statement generation method can further include: obtaining training data, the training data including a sample natural language text and a corresponding expected structured query statement; obtaining a sample related field, a sample related data table, a sample prompt information and a sample question type information of the sample natural language text; performing feature interaction on the sample natural language text, the sample related field, the sample related data table, the sample prompt information and the sample question type information by the basic module to obtain sample feature interaction information; for each sub-module of the adaptive module, performing feature interaction on the sample natural language text, the sample related field, the sample related data table, the sample prompt information and the sample question type information by the sub-module to obtain sample sub-feature interaction information output by the sub-module; performing fusion processing on the sample feature interaction information and the sample sub-feature interaction information output by each sub-module by the integration module to obtain an actual structured query statement of the sample natural language text; and adjusting parameters of each sub-module in the adaptive module according to the actual structured query statement and the expected structured query statement to obtain a trained query statement generation model.

[0207] The sample related field, the sample related data table, the sample prompt information and the sample question type information of the sample natural language text can be obtained in the same manner as the task-related field, the task-related data table, the prompt information and the question type information of the natural language text.

[0208] The basic module can be a pre-trained model, and the parameters of the basic module do not need to be adjusted in this embodiment.

[0209] The training process can be to calculate the similarity between the actual structured query statement and the expected structured query statement, and then use the back propagation method to adjust the parameters in the sub-module, and based on the actual structured query statement and the expected structured query statement, optimize the parameters of the sub-module in the query statement generation model, so that the actual structured query language tends to the expected structured query language.

[0210] Specifically, the sample natural language text has multiple sample question type information, and for a sub-module corresponding to a question type information, the parameters of the sub-module can be adjusted according to the actual structured query statement and the expected structured query statement of the sample natural language text under the corresponding sample question type information.

[0211] In specific embodiments, as shown in FIG. 1c, the pre-trained model can be first fine-tuned based on task data to obtain a fine-tuned pre-trained model. The fine-tuned pre-trained model can be used as a basic module of the query statement generation model in the above embodiments. An adaptive module is added to the bypass of the basic module. The adaptive module can include sub-modules corresponding to each preset question type information, such as a sub-module for simple problems, a sub-module for medium problems, and a sub-module for difficult problems. Each sub-module has a corresponding dimension reduction matrix A and a dimension increase matrix B. Through the LoRA fine-tuning algorithm, only the dimension reduction matrix and the dimension increase matrix of each sub-module in the adaptive module can be fine-tuned. During the low-parameter fine-tuning of the adaptive module, the parameters (i.e., pre-training parameters) W of the basic module are frozen, so that after fine-tuning the parameters in the adaptive module, a new model can be obtained, and the new model can be applied to the inference process. d*d

[0212] Full-parameter fine-tuning is a process of adjusting and optimizing all parameters of a pre-trained model based on task data. This way makes the pre-trained model more suitable for specific task requirements. In this application, the fine-tuned pre-trained model can be used as a basic module of the query statement generation model. Low-parameter fine-tuning is a fine-tuning method that adjusts a small number of parameters to adapt the model to a specific task without adjusting a large number of parameters. For example, LoRA (Low-Rank Adaptation), QLORA, adapter tuning, prefix-tuning, etc. can reduce the number of parameters involved in training and reduce training costs.

[0213] Prefix tuning is a low-parameter fine-tuning method. By obtaining training data and a learnable feedforward vector corresponding to each preset question type information, the sample natural language text, the sample associated field, the sample associated data table, the sample prompt information, the sample question type information, and the learnable feedforward vector are input into the query statement generation model for attention feature interaction. The parameters of the learnable feedforward vector are adjusted according to the generated actual structured query statement and the expected structured query statement, and other parameters of the model are not adjusted.

[0214] ​Optionally, in the embodiment, the query statement generation model comprises a feedforward module, an adaptive module and a normalization module; the adaptive module comprises sub-modules corresponding to respective preset question type information, and each sub-module comprises a down-sampling layer and an up-sampling layer; the step of "generating the structured query statement of the rewritten natural language text based on the task association field, the task association data table, the prompt information and the question type information through the query statement generation model" can comprise: obtaining feature interaction information by performing feature interaction on the rewritten natural language text, the task association field, the task association data table, the prompt information and the question type information through the feedforward module; performing down-sampling processing on the feature interaction information through the down-sampling layer of each sub-module of the adaptive module to obtain down-sampled feature information output by the down-sampling layer of each sub-module; performing up-sampling processing on the down-sampled feature information output by the corresponding down-sampling layer through the up-sampling layer of each sub-module of the adaptive module to obtain up-sampled feature information output by the up-sampling layer of each sub-module; and performing normalization processing on the up-sampled feature information output by the up-sampling layer of each sub-module through the normalization module to obtain the structured query statement of the rewritten natural language text.

[0215] The query statement generation model can be an Adapter tuning model, which is a subnetwork with a smaller number of parameters added to a pre-trained Bert model. During fine-tuning, only the parameters of the subnetwork are updated while the other parameters remain unchanged. The main architecture of the Adapter is shown in FIG. 1d. Specifically, an Adapter module (i.e., the adaptive module, as shown in the left structure of FIG. 1d) can be added to each layer (or some layers) of the pre-trained model. During fine-tuning, the pre-trained model is frozen, and the Adapter module learns the knowledge of specific downstream tasks. Each Adapter module consists of two feedforward sublayers. The first feedforward sublayer takes the output of the Transformer block as input, projects the original input dimension d to m, and limits the parameter quantity of the Adapter module by controlling the size of m. In general, m << d. In the output stage, the input dimension is restored by the second feedforward sublayer, and m is projected back to d as the output of the Adapter module (as shown in the right structure of FIG. 1d). By adding the Adapter module, an easily expandable downstream model is generated. Whenever a new downstream task appears, the Adapter module is added to avoid the problem of full model fine-tuning and catastrophic forgetting. The Adapter method does not need to fine-tune all the parameters of the pre-trained model. By introducing a small number of parameters specific to a particular task, the knowledge about the task is stored, and the computational requirement for model fine-tuning is reduced.

[0216] The feedforward module is part of the query statement generation model, and features of the rewritten natural language text, the task-related field, the task-related data table, the prompt information, and the question type information are interacted to obtain feature interaction information, thereby providing a basis for generating a structured query statement.

[0217] The normalization module is part of the query statement generation model, and the up-sampled feature information output by the up-sampling layer of each submodule in the adaptive module is normalized to obtain a structured query statement of the rewritten natural language text.

[0218] Optionally, before the step of "obtaining feature interaction information by the feedforward module from the rewritten natural language text, the task-related field, the task-related data table, the prompt information, and the question type information", the structured query statement generation method can further include: obtaining training data, the training data including a sample natural language text and a corresponding expected structured query statement; obtaining a sample-related field, a sample-related data table, sample prompt information, and sample question type information of the sample natural language text; obtaining sample feature interaction information by the feedforward module from the sample natural language text, the sample-related field, the sample-related data table, the sample prompt information, and the sample question type information; performing down-sampling processing on the sample feature interaction information by the down-sampling layer of each submodule in the adaptive module to obtain down-sampled feature information output by the down-sampling layer of each submodule; performing up-sampling processing on the down-sampled feature information output by the corresponding down-sampling layer by the up-sampling layer of each submodule in the adaptive module to obtain up-sampled feature information output by the up-sampling layer of each submodule; performing normalization processing on the up-sampled feature information output by the up-sampling layer of each submodule by the normalization module to obtain an actual structured query statement of the sample natural language text; and adjusting parameters of each submodule in the adaptive module according to the actual structured query statement and the expected structured query statement to obtain a trained query statement generation model. The sample-related field, the sample-related data table, the sample prompt information, and the sample question type information of the sample natural language text can be obtained in the same manner as the task-related field, the task-related data table, the prompt information, and the question type information of the natural language text.

[0219] The feedforward module and the normalization module belong to the original pre-trained model, and the adaptive module is added based on the original pre-trained model. The adaptive module can be added after the feedforward module and before the normalization module. Specifically, the parameters of the feedforward module and the normalization module do not need to be adjusted, and only the parameters of the adaptive module need to be adjusted.

[0220] The training process can be calculating the similarity between the actual structured query statement and the expected structured query statement, and then adjusting the parameters in the sub-model using a back propagation method, optimizing the parameters of the sub-modules in the query statement generation model based on the actual structured query statement and the expected structured query statement, so that the actual structured query language approaches the expected structured query language.

[0221] Specifically, the sample natural language text has multiple sample question type information, and for a sub-module corresponding to a question type information, the parameters of the sub-module can be adjusted according to the actual structured query statement and the expected structured query statement of the sample natural language text under the corresponding sample question type information.

[0222] Optionally, in the embodiment, the step of "generating the structured query statement of the rewritten natural language text based on the task association field, the task association data table, the prompt information and the question type information through the query statement generation model" can include: obtaining a pre-feed vector corresponding to each preset question type information; performing attention feature interaction on the rewritten natural language text, the task association field, the task association data table, the prompt information, the question type information and the pre-feed vector corresponding to each preset question type information through the query statement generation model to obtain feature interaction information; and generating the structured query statement of the rewritten natural language text according to the feature interaction information through the query statement generation model.

[0223] The pre-feed vector corresponding to each preset question type information is obtained through a large amount of sample learning.

[0224] Optionally, before the step of "obtaining the pre-feed vector corresponding to each preset question type information", the structured query statement generation method can further include: obtaining training data and a learnable pre-feed vector corresponding to each preset question type information, the training data including sample natural language text and its corresponding expected structured query statement; obtaining sample association fields, sample association data tables, sample prompt information and sample question type information of the sample natural language text; performing attention feature interaction on the sample natural language text, the sample association fields, the sample association data tables, the sample prompt information, the sample question type information and the learnable pre-feed vector corresponding to each preset question type information through the query statement generation model to obtain sample feature interaction information; generating an actual structured query statement of the sample natural language text according to the sample feature interaction information through the query statement generation model; and adjusting each learnable pre-feed vector according to the actual structured query statement and the expected structured query statement to obtain a learned pre-feed vector.

[0225] The learnable feedforward vector is a vector constructed through learning of a large number of samples, and in the query statement generation model, the learnable feedforward vector interacts with the rewritten natural language text, the task-associated field, the task-associated data table, the prompt information and the question type information to generate a structured query statement. The learnable feedforward vector can be learned and adjusted through a prefix tuning method.

[0226] The embodiment can realize low-parameter fine-tuning through a prefix tuning method. Training data including sample natural language texts and corresponding expected structured query statements can be obtained. Sample associated fields, sample associated data tables, sample prompt information and sample question type information of the sample natural language texts can be obtained. The sample natural language texts, the sample associated fields, the sample associated data tables, the sample prompt information, the sample question type information and learnable feedforward vectors corresponding to each preset question type information are input into a query statement generation model to interact with each other to obtain sample feature interaction information. An actual structured query statement of the sample natural language text is generated according to the sample feature interaction information. Similarity between the actual structured query statement and the expected structured query statement is calculated, for example, using an edit distance, a cosine similarity or the like. A loss function is calculated according to the similarity, for example, a cross-entropy loss function. An optimization algorithm, for example, a stochastic gradient descent algorithm, is used to optimize the loss function to update parameters of the learnable feedforward vectors. During the updating process, other parameters of the query statement generation model are not adjusted, and only the feedforward vectors of each preset question type information are learned, that is, other parameters are frozen, and only the feedforward vectors are fine-tuned. The learning process is iterated until the loss function converges, and the learned feedforward vectors are obtained.

[0227] The sample associated fields, the sample associated data tables, the sample prompt information and the sample question type information of the sample natural language texts can be obtained in the same manner as the task-associated field, the task-associated data table, the prompt information and the question type information of the natural language text.

[0228] Specifically, in the embodiment, the parameters of the query statement generation model can not be adjusted, and only the feedforward vectors of each preset question type information can be learned, that is, other parameters are frozen, and only the feedforward vectors are fine-tuned.

[0229] The learning process of the feedforward vector can be calculating similarity between the actual structured query statement and the expected structured query statement, adjusting the feedforward vector, optimizing the feedforward vector based on the actual structured query statement and the expected structured query statement, and making the actual structured query statement approach the expected structured query statement.

[0230] Specifically, the sample natural language text has multiple sample problem type information, and for a feedforward vector corresponding to a problem type information, the feedforward vector corresponding to the problem type information can be learned according to the actual structured query statement and the expected structured query statement of the sample natural language text under the corresponding sample problem type information.

[0231] The application proposes a text2SQL architecture combining a new fine-tuning paradigm of a model and an agent workflow paradigm, which can effectively improve the performance of the task and greatly enhance the accuracy of SQL generation. The specific agent steps can include: information understanding (entity disassembly, general problem rewriting), information recall layer, information sorting, information optimization (field accuracy, problem rewriting, field pre-check), retrieval enhancement layer, SQL generation and execution optimization (problem classification (bert classification), LORA+MOE fine-tuning model, SQL check (check+execution correction)).

[0232] In the LORA+MOE fine-tuning paradigm design adapted to the text2SQL fine-tuning task, the problems encountered in text2sql are classified according to different problem asking methods and SQL difficulty levels; different problem classifications use different lora training of the same base, and the MOE algorithm is combined to further improve the accuracy of SQL generation.

[0233] It can be understood that the fine-tuning paradigm of the application can also be QLORA, adapter tuning, prefix-tuning and other common fine-tuning methods.

[0234] In a specific business scenario, the structured query statement generation system provided by the application can include an information understanding layer, an information recall and sorting layer, an information optimization layer, a retrieval enhancement layer, and a structured query statement generation and optimization execution layer, as shown in FIG. 1e.

[0235] The information understanding layer is a level of the structured query statement generation system, which disassembles and rewrites the natural language problem text proposed by the user into a more understandable form, and at the same time, performs key element entity recognition to provide necessary information for subsequent information recall and SQL generation.

[0236] The information recall and sorting layer is a level of the structured query statement generation system, which performs similarity retrieval and Elasticsearch retrieval multi-path recall table and field information in a large number of database tables according to key entities and problems, and determines the most relevant database table and related field to the problem proposed by the user.

[0237] The information optimization layer is a level of the structured query statement generation system, including information accuracy, question rewriting, and field pre-checking. It further refines the table-related information obtained from the information recall sorting layer, eliminates irrelevant information, and selects the most relevant table and field information to the user's question.

[0238] The retrieval enhancement layer is a level of the structured query statement generation system. It retrieves relevant prompt information from the retrieval library (such as time condition library, join table condition library, calculation logic library, Fewshot library, etc.) according to the user's question, assembles these information in a certain format, submits it to the prompt of the SQL generation model, and can also add personalized recommendation function to improve user experience.

[0239] The structured query statement generation and optimization execution layer is a level of the structured query statement generation system, including a question classification module, a SQL model module, and a result correction module. It classifies the question, generates a structured query statement through the SQL model module, and corrects the generated statement to improve the accuracy of data query.

[0240] Among them, for the information understanding layer, the natural language question text proposed by the user is disassembled and rewritten into a general question to make the question easier to understand. Assuming that the natural language question text proposed by the user is "How many new member users has a certain video app added in the last 7 days?", in order to ensure the completeness of the information in the recall layer and some additional information necessary for the subsequent generation of SQL, the question can be first identified for key element entities: such as time condition, core element condition. An example of entity recognition of the user's question is as follows:

[0241] Core element condition: member user;

[0242] Time condition: last 7 days.

[0243] Through question rewriting, it can be rewritten by LLM (Large Language Model) or other self-developed models.

[0244] For the information recall sorting layer, this layer needs to determine the most relevant database table and related field from a large number of database tables to the question proposed by the user. On the recall side, similar degree retrieval and Elasticsearch retrieval can be performed according to the key entities and questions to retrieve table and field information.

[0245] Among them, the information optimization layer can include information accuracy, problem rewriting, and field pre-check three parts. For the above information recall ranking layer, a relatively small number of table related information is obtained. However, for the SQL generation task, there are still many irrelevant information that will interfere with the learning of the model, so the information optimization layer can further reduce the irrelevant information and select the table and field information most related to the user question. The information accuracy module of the information optimization layer can be used to simplify the table and field.

[0246] Among them, the retrieval enhancement layer can retrieve relevant prompt information according to the user question, assemble these information according to a certain format, and finally submit the prompt to the SQL generation model. Here, the retrieval library can include time condition library, join table condition library, calculation logic library, Fewshot library (i.e. few-shot retrieval library), etc. For the use of each retrieval library, please refer to the above embodiments.

[0247] Optionally, personalized recommendation function can also be added to the retrieval enhancement layer to improve user experience. With the authorization of the user, the personalized recommendation function can analyze the user's historical questions and behavior data, and the system can learn the user's interests and needs to provide more accurate and personalized pre-prompts for the user.

[0248] For the structured query statement generation and optimization execution layer, it can include a question classification module, a SQL model module, and a result correction module.

[0249] The question classification module can classify the question according to the question method paradigm and the SQL difficulty. Among them, the SQL difficulty classification can be divided into simple questions (without join table and nested), nested questions (only need nested subquery), join table questions (only need join table) and join table nested questions according to whether it needs to join table and nest. After the question classification module, the question will be classified into a specific question method paradigm and SQL difficulty classification.

[0250] The SQL model module is the core of the database query generation system, which is responsible for converting natural language questions into SQL query statements. This layer mainly trains the SQL generation model, and the main training methods can be SFT (Supervised Fine-Tuning, supervised fine-tuning), full parameter fine-tuning, etc.

[0251] Among them, the SFT dataset required for training can be constructed according to the sample question, related table information, and join table conditions obtained in the previous steps. The expected SQL of the sample question can be manually annotated, or the SQL statement can be automatically generated by developing tools or using existing libraries to reduce human intervention and improve efficiency. As shown below, it is a sample of data input model (the assembly form is not unique):

[0252] ###Question

[0253] [Question]

[0254] ###Schema

[0255] [Schema]

[0256] [Join Condition]

[0257] ###Output

[0258] [sql]

[0259] To improve the accuracy of different categories of problems, the embodiment designs a fine-tuning scheme of LORA (Low-Rank Adaptation) + MOE (Mixture of Experts), trains an LORA module for data of different problem categories, and combines the LORA modules at a high level using the MOE algorithm to improve the SQL generation accuracy of different problem categories.

[0260] The LORA module is a lightweight and efficient model adaptation method that introduces low-rank matrices into key parts of the model to quickly adapt to specific tasks. In this embodiment, an independent LORA module is set up for each problem category, which focuses on learning features and patterns related to the category.

[0261] The MOE algorithm is an effective model integration technique that combines the outputs of multiple expert models to improve overall performance. In this embodiment, each LORA module can be considered an expert, and the LORA modules are combined at a high level using the MOE algorithm to implement multiple tasks through multi-LORA + MOE, thereby achieving accurate identification of different problem categories. Compared with the scheme of directly training multiple base models, the scheme provided in this application only requires one base model and multiple LORA modules, which can reduce model resource consumption. The base model can be a pre-trained model.

[0262] After training the model, the automatic evaluation module can also evaluate the matching accuracy and execution accuracy of the SQL generated by the model.

[0263] The result correction module optimizes the SQL generated in the previous step and determines whether the SQL can solve the user's problem. Since there is still a small probability that the result generated by the large model cannot solve the user's problem, such as the generated SQL being unable to execute or the generated SQL having subtle conditions, the result correction module can be used to check the SQL to improve the accuracy of data query using the SQL statement.

[0264] The structured query statement generation method of the application can be used in a Text2SQL application. Through the Text2SQL application, natural language text input by a user can be converted into a corresponding SQL query statement and a simple data analysis result, thereby reducing the technical threshold of product operation and data analysis personnel in data analysis work. Specifically, in the SQL generation task, the scheme using the agent workflow is compared with the existing related scheme. The accuracy of the scheme on the business data set is improved from 50% to 70%+.

[0265] As can be seen from the above, the embodiment can determine natural language text representing a query task, and determine a preset database, the preset database including at least one data table, each data table including at least one field; based on the natural language text, determine at least one task-associated field and at least one task-associated data table associated with the natural language text from the preset database; rewrite the natural language text according to the task-associated field to obtain rewritten natural language text; obtain prompt information for the rewritten natural language text, the prompt information being used to guide a query statement generation model to generate a structured query statement; perform question classification on the rewritten natural language text to obtain question type information of the rewritten natural language text; and generate, by the query statement generation model, a structured query statement of the rewritten natural language text based on the task-associated field, the task-associated data table, the prompt information, and the question type information.

[0266] The application can first determine corresponding task-associated fields and task-associated data tables based on the natural language text, and rewrite the natural language text. Through the rewriting, the natural language text can be more easily understood by the model. Finally, the task-associated fields, the task-associated data tables, corresponding prompt information, and question type information are combined to generate a structured query statement. In this way, the generation task of the query statement is more comprehensively considered, a more detailed task process is provided, and the generation ability of the model is avoided, which can greatly improve the accuracy of the structured query statement generation.

[0267] According to the method described in the foregoing embodiments, the following will be further described in detail with the example that the structured query statement generation apparatus is specifically integrated in a server.

[0268] The embodiment of the application provides a structured query statement generation method, as shown in FIG. 2. The specific process of the structured query statement generation method can be as follows:

[0269] 201. The server determines natural language text representing a query task, and determines a preset database, the preset database including at least one data table, each data table including at least one field.

[0270] The natural language text is specifically a natural language question text proposed by a user. The preset database is an electronic warehouse for organizing, storing, and managing related business data, and the user can perform operations such as adding, querying, updating, and deleting data in the preset database.

[0271] 202. The server recalls at least one candidate field and at least one candidate data table from the preset database based on the natural language text.

[0272] The candidate field is a field that may be related to the natural language text and is recalled from the preset database based on the natural language text representing the query task, and the task-related field will be determined by calculating the similarity between the natural language text and the candidate field later. The candidate data table is a data table that may be related to the natural language text and is recalled from the preset database based on the natural language text representing the query task, and the task-related data table will be determined by calculating the similarity between the natural language text and the candidate data table later.

[0273] Optionally, the candidate field and the candidate data table can be recalled based on the natural language text. There are various ways to recall the candidate field and the candidate data table, such as through entity similarity, question similarity, or Elasticsearch retrieval. In specific scenarios, key entities and questions can be used for similarity retrieval and Elasticsearch retrieval for multi-path recall.

[0274] Optionally, in this embodiment, the step of "recalling at least one candidate field and at least one candidate data table from the preset database based on the natural language text" can include: performing entity recognition processing on the natural language text to obtain an entity set corresponding to the natural language text; recalling at least one candidate field from the preset database based on the similarity between the entities in the entity set and the fields in the preset database; and recalling at least one candidate data table from the preset database based on the similarity between the entities in the entity set and the table information of the data tables in the preset database.

[0275] In this embodiment, the candidate field and the candidate data table can be recalled through entity similarity. Entity similarity recall specifically refers to recalling by calculating the similarity between entities according to the entities contained in the natural language question text proposed by the user. The specific implementation methods of entity similarity recall include but are not limited to: word vector-based similarity calculation recall and knowledge graph-based similarity calculation recall.

[0276] Optionally, in the embodiment, the step of "recalling at least one candidate field and at least one candidate data table from the preset database based on the natural language text" can include: calculating a matching degree between the natural language text and each preset question, each preset question being associated with at least one data table in the preset database; determining a similar question from the preset questions according to the matching degree; determining the data table associated with the similar question as a candidate data table; and recalling at least one candidate field from the preset database based on the natural language text.

[0277] In the embodiment, the candidate data table can be recalled by question similarity. Specifically, the preset database can further include preset questions, each of which is associated with a data table. The embodiment can recall the data table by calculating the similarity between the natural language question text proposed by the user and each preset question.

[0278] In the embodiment, the preset question with a matching degree greater than a preset matching degree can be determined as the similar question. Alternatively, the preset questions can be sorted according to the matching degree, for example, sorted from large to small to obtain sorted questions, and the first K preset questions in the sorted questions can be determined as the similar questions.

[0279] Optionally, in the embodiment, the step of "recalling at least one candidate field and at least one candidate data table from the preset database based on the natural language text" can include: performing data analysis on the data tables in the preset database to determine the key information of the data tables; generating a retrieval index of the data tables according to the key information of the data tables; creating a retrieval index library based on the retrieval index, the retrieval index library including a mapping relationship between the retrieval index and the data tables; and performing data table and field searching in the retrieval index library based on the natural language text to obtain at least one candidate data table and candidate field.

[0280] In the embodiment, the candidate data table and the candidate field can be recalled by question Elasticsearch retrieval. Specifically, the question Elasticsearch retrieval refers to performing processing such as tokenization and stop word removal on the natural language question text proposed by the user to obtain a query statement, performing retrieval through an Elasticsearch search engine, and returning related tables and fields.

[0281] In the embodiment, the key information of the data table can be used as the retrieval index of the data table, and then the retrieval index library can be created according to the retrieval index, and one retrieval index can correspond to one or more data tables. Specifically, the natural language question text can be processed such as tokenization and stop word removal to obtain a query statement for retrieval.

[0282] 203、The server calculates the similarity of each candidate field and each candidate data table with the natural language text respectively, to determine the task-related field from the candidate fields and the task-related data table from the candidate data tables according to the similarity.

[0283] Optionally, in the embodiment, the step of "calculating the similarity of each candidate field and each candidate data table with the natural language text respectively, to determine the task-related field from the candidate fields and the task-related data table from the candidate data tables according to the similarity" can include: filtering the candidate data tables according to a preset data strategy to obtain filtered candidate data tables; detecting the relevance of each candidate field and each filtered candidate data table with the natural language text through a first preset model, to determine the task-related field from each candidate field and the associated data table from each filtered candidate data table; fusing the table identifier information and the table field of the associated data table to obtain table information; fusing the table information and the entity of the natural language text through a second preset model to obtain fused information; extracting the feature information of the fused information, to predict the similarity of the associated data table and the natural language text based on the feature information; and determining the task-related data table from the associated data table according to the similarity.

[0284] The preset data strategy can be set according to actual business requirements, for example, the preset data strategy can be some field rules. Specifically, the candidate data table includes related data tables of App (Application) and Pc (Personal Computer) ends, and the preset data strategy can be to filter out the data tables related to the "pc end".

[0285] 204、The server rewrites the natural language text according to the task-related field to obtain a rewritten natural language text.

[0286] In the embodiment, the natural language text can be rewritten so that the natural language text can be better understood by the model. Specifically, the entity in the natural language text can be rewritten based on the task-related field through a question rewriting model, to rewrite the natural language text into a style that is easier for the model to understand, and to improve the accuracy of the final SQL generation.

[0287] Optionally, before the step of "rewriting the natural language text according to the task association field to obtain a rewritten natural language text", the embodiment can further include: filtering the task association field based on the detection result to obtain a filtered task association field.

[0288] In the embodiment, after the data table and the field are recalled and sorted, the information optimization model can be used to further perform information simplification processing on the data table and the field, so as to delete irrelevant information, select the table and field information most relevant to the natural language question text proposed by the user, and reduce the interference of irrelevant information on the query statement generation model.

[0289] 205. The server obtains prompt information for the rewritten natural language text, and the prompt information is used to guide the query statement generation model to generate a structured query statement.

[0290] In some embodiments, the prompt information can be generated based on the rewritten natural language text by using a neural network model. In other embodiments, the prompt information can also be selected from a preset prompt library based on the rewritten natural language text. The preset prompt library can include a large amount of preset prompt information.

[0291] Optionally, in the embodiment, the step of "obtaining prompt information for the rewritten natural language text" can include: obtaining a query statement generation model; selecting a target prompt library from each preset prompt library according to the rewritten natural language text, and determining the prompt information of the rewritten natural language text according to the correspondence between the information in the target prompt library and the rewritten natural language text.

[0292] In the embodiment, different preset prompt libraries can include different types of prompt information. The corresponding target prompt library can be selected from each preset prompt library according to the information such as entities contained in the rewritten natural language text, and then appropriate prompt information can be selected from the target prompt library.

[0293] 206. The server classifies the rewritten natural language text to obtain question type information of the rewritten natural language text.

[0294] In the embodiment, the natural language question text can be classified according to the question asking method paradigm and the SQL difficulty to obtain corresponding question type information. The question type information can help the query statement generation model to better generate the SQL query statement.

[0295] 207. The server generates a structured query statement of the rewritten natural language text based on the task association field, the task association data table, the prompt information, and the question type information through a query statement generation model.

[0296] Optionally, after the step of "generating a structured query statement of the rewritten natural language text based on the task association field, the task association data table, the prompt information, and the question type information through a query statement generation model", the structured query statement generation method can further include: verifying the structured query statement; and when the verification result is passed, querying in a preset database based on the structured query statement to obtain a query result corresponding to the natural language text.

[0297] In this embodiment, after the query statement generation model generates the structured query statement of the rewritten natural language text, the structured query statement of the rewritten natural language text can be further verified, and the accuracy of data query using the SQL query statement can be improved through the verification.

[0298] Optionally, in this embodiment, the query statement generation model includes a basic module, an adaptive module, and an integration module, the adaptive module includes a sub-module corresponding to each preset question type information; the step of "generating a structured query statement of the rewritten natural language text based on the task association field, the task association data table, the prompt information, and the question type information through a query statement generation model" can include: through the basic module, performing feature interaction on the rewritten natural language text, the task association field, the task association data table, the prompt information, and the question type information to obtain feature interaction information; for each sub-module, performing feature interaction on the rewritten natural language text, the task association field, the task association data table, the prompt information, and the question type information through the sub-module to obtain sub-feature interaction information output by the sub-module; and through the integration module, performing fusion processing on the feature interaction information and the sub-feature interaction information output by each sub-module to obtain the structured query statement of the rewritten natural language text.

[0299] Each sub-module in the adaptive module can be a LORA (Low-Rank Adaptation) module, and one LORA module is one expert model.

[0300] The integration module can be a MOE (Mixture of Experts) module, which combines the outputs of multiple expert models through weighted combination to improve the overall performance.

[0301] Optionally, before the step of "performing feature interaction on the rewritten natural language text, the task-related field, the task-related data table, the prompt information and the question type information by the basic module to obtain feature interaction information", the structured query statement generation method can further include: obtaining training data, the training data including a sample natural language text and a corresponding expected structured query statement; obtaining a sample related field, a sample related data table, a sample prompt information and a sample question type information of the sample natural language text; performing feature interaction on the sample natural language text, the sample related field, the sample related data table, the sample prompt information and the sample question type information by the basic module to obtain sample feature interaction information; for each sub-module of the adaptive module, performing feature interaction on the sample natural language text, the sample related field, the sample related data table, the sample prompt information and the sample question type information by the sub-module to obtain sample sub-feature interaction information output by the sub-module; performing fusion processing on the sample feature interaction information and the sample sub-feature interaction information output by each sub-module by the integration module to obtain an actual structured query statement of the sample natural language text; and adjusting parameters of each sub-module in the adaptive module according to the actual structured query statement and the expected structured query statement to obtain a trained query statement generation model.

[0302] The sample related field, the sample related data table, the sample prompt information and the sample question type information of the sample natural language text can be obtained in the same manner as the task-related field, the task-related data table, the prompt information and the question type information of the natural language text.

[0303] The basic module can be a pre-trained model, and the parameters of the basic module do not need to be adjusted in this embodiment.

[0304] The training process can be to calculate the similarity between the actual structured query statement and the expected structured query statement, and then use the back propagation method to adjust the parameters in the sub-module, and based on the actual structured query statement and the expected structured query statement, optimize the parameters of the sub-module in the query statement generation model, so that the actual structured query language tends to the expected structured query language.

[0305] Specifically, the sample natural language text has multiple sample question type information, and for a sub-module corresponding to a question type information, the parameters of the sub-module can be adjusted according to the actual structured query statement and the expected structured query statement of the sample natural language text under the corresponding sample question type information.

[0306] From the above, the embodiment can determine the natural language text representing the query task through the server, and determine a preset database, the preset database including at least one data table, each data table including at least one field; based on the natural language text, recall at least one candidate field and at least one candidate data table from the preset database; calculate the similarity of each candidate field and each candidate data table with the natural language text, to determine a task-associated field from the candidate fields and a task-associated data table from the candidate data tables according to the similarity; rewrite the natural language text according to the task-associated field to obtain a rewritten natural language text; obtain prompt information for the rewritten natural language text, the prompt information being used to guide a structured query statement generation model to generate a structured query statement; perform question classification on the rewritten natural language text to obtain question type information of the rewritten natural language text; and generate a structured query statement of the rewritten natural language text through the query statement generation model based on the task-associated field, the task-associated data table, the prompt information, and the question type information.

[0307] The application can first determine the corresponding task-associated field and task-associated data table based on the natural language text, and rewrite the natural language text, so that the natural language text is more easily understood by the model, and finally the structured query statement is generated in combination with the task-associated field, the task-associated data table, the corresponding prompt information, and the question type information, which more comprehensively considers the generation task of the structured query statement, provides a more detailed task process, avoids relying only on the generation ability of the model, and can greatly improve the accuracy of the structured query statement generation.

[0308] In order to better implement the above method, the embodiment of the application further provides a structured query statement generation device, as shown in FIG. 3, which can include a first determination unit 301, a second determination unit 302, a rewriting unit 303, an obtaining unit 304, a question classification unit 305, and a generation unit 306, as follows:

[0309] (1) The first determination unit 301;

[0310] The first determination unit is configured to determine a natural language text representing a query task, and determine a preset database, the preset database including at least one data table, each data table including at least one field.

[0311] (2) The second determination unit 302;

[0312] The second determination unit is configured to determine at least one task-associated field and at least one task-associated data table associated with the natural language text from the preset database based on the natural language text.

[0313] Optionally, in some embodiments of the present application, the second determining unit can include a recalling subunit and a determining subunit, as follows:

[0314] The recalling subunit is configured to recall at least one candidate field and at least one candidate data table from the preset database based on the natural language text.

[0315] The determining subunit is configured to calculate the similarity between each candidate field and the natural language text and the similarity between each candidate data table and the natural language text, and determine the task-related field from the candidate fields and the task-related data table from the candidate data tables according to the similarities.

[0316] Optionally, in some embodiments of the present application, the recalling subunit can be specifically configured to perform entity recognition processing on the natural language text to obtain an entity set corresponding to the natural language text, recall at least one candidate field from the preset database based on the similarity between the entities in the entity set and the fields in the preset database, and recall at least one candidate data table from the preset database based on the similarity between the entities in the entity set and the table information of the data tables in the preset database.

[0317] Optionally, in some embodiments of the present application, the recalling subunit can be specifically configured to calculate the matching degree between the natural language text and each preset question, each preset question being associated with at least one data table in the preset database, determine a similar question from the preset questions according to the matching degree, determine the data table associated with the similar question as the candidate data table, and recall at least one candidate field from the preset database based on the natural language text.

[0318] Optionally, in some embodiments of the present application, the recalling subunit can be specifically configured to perform data analysis on the data tables in the preset database to determine the key information of the data tables, generate a retrieval index of the data tables according to the key information of the data tables, create a retrieval index library based on the retrieval index, the retrieval index library including the mapping relationship between the retrieval index and the data tables, and perform data table and field searching in the retrieval index library based on the natural language text to obtain at least one candidate data table and candidate field.

[0319] Optionally, in some embodiments of the present application, the determining subunit can be specifically configured to filter the candidate data tables according to a preset data strategy to obtain filtered candidate data tables; detect, by using a first preset model, the relevance of each candidate field and each filtered candidate data table to the natural language text, to determine a task-related field from each candidate field and determine a related data table from each filtered candidate data table; fuse the table identifier information and the table field of the related data table to obtain table information; fuse, by using a second preset model, the table information and the entity of the natural language text to obtain fused information; extract feature information of the fused information, and predict the similarity between the related data table and the natural language text based on the feature information; and determine a task-related data table from the related data table according to the similarity.

[0320] (3) the rewriting unit 303;

[0321] The rewriting unit is configured to rewrite the natural language text according to the task-related field to obtain a rewritten natural language text.

[0322] Optionally, in some embodiments of the present application, the structured query statement generation apparatus further includes an information optimization unit, a detection unit and a filtering unit, as follows:

[0323] The information optimization unit is configured to filter, by using an information optimization model, the task-related data table and the task-related field based on the natural language text to obtain a filtered task-related data table and a filtered task-related field.

[0324] The detection unit is configured to detect, based on the filtered task-related data table, whether the filtered task-related field can perform the structured query task of the natural language text.

[0325] The filtering unit is configured to filter, based on the detection result, the filtered task-related field to obtain a filtered task-related field.

[0326] (4) the obtaining unit 304;

[0327] The obtaining unit is configured to obtain prompt information for the rewritten natural language text, the prompt information being used to guide the query statement generation model to generate a structured query statement.

[0328] Optionally, in some embodiments of the present application, the obtaining unit can include a model obtaining subunit and a prompt determining subunit, as follows:

[0329] The model obtaining subunit is configured to obtain the query statement generation model.

[0330] The prompt determining subunit is configured to select a target prompt library from each preset prompt library according to the rewritten natural language text, and determine the prompt information of the rewritten natural language text according to a correspondence between information in the target prompt library and the rewritten natural language text.

[0331] Optionally, in some embodiments of the present application, the preset prompt library includes a preset time condition library, a preset join table condition library, a preset calculation logic library and a preset few-shot library; the prompt determining subunit can be specifically configured to: when the rewritten natural language text contains a time entity, select the preset time condition library as the target prompt library, so as to retrieve time partition condition information corresponding to the time entity from the target prompt library, and generate the prompt information according to the time partition condition information; when the rewritten natural language text is related to a data table connection, select the preset join table condition library as the target prompt library, so as to retrieve join table condition information corresponding to the task-associated data table from the target prompt library, and generate the prompt information based on the join table condition information; when the rewritten natural language text contains a calculation index, select the preset calculation logic library as the target prompt library, so as to retrieve calculation logic information corresponding to the calculation index from the target prompt library, and generate the prompt information according to the calculation logic information; when the rewritten natural language text is related to a few-shot learning problem, select the preset few-shot library as the target prompt library, so as to retrieve a structured query generation strategy corresponding to the few-shot learning problem from the target prompt library, and generate the prompt information based on the structured query generation strategy.

[0332] (5) the question classification unit 305;

[0333] The question classification unit is configured to perform question classification on the rewritten natural language text to obtain question type information of the rewritten natural language text.

[0334] Optionally, in some embodiments of the present application, the question classification unit can include a paradigm determining subunit, a generation difficulty determining subunit and a type determining subunit, as follows:

[0335] The paradigm determining subunit is configured to determine a question paradigm corresponding to the rewritten natural language text.

[0336] The generation difficulty determining subunit is configured to determine a generation difficulty level of a query statement corresponding to the task-associated data table according to data structure association information between the task-associated data tables.

[0337] The type determining subunit is configured to determine the question type information of the rewritten natural language text from each preset question type information based on the question paradigm and the generation difficulty level, where each preset question type information corresponds to one preset question paradigm and one preset generation difficulty level.

[0338] Optionally, in some embodiments of the present application, the difficulty determination subunit can be specifically configured to perform nested table detection and cross table detection on the task-related data table; and determine the generation difficulty level of the query statement corresponding to the task-related data table according to the detection result.

[0339] (6) the generation unit 306;

[0340] The generation unit is configured to generate, by a query statement generation model, a structured query statement of the rewritten natural language text based on the task-related field, the task-related data table, the prompt information, and the question type information.

[0341] Optionally, in some embodiments of the present application, the structured query statement generation apparatus can further include a verification unit and a query unit, as follows:

[0342] The verification unit is configured to verify the structured query statement;

[0343] The query unit is configured to, when the verification result is passed, query in a preset database based on the structured query statement to obtain a query result corresponding to the natural language text.

[0344] Optionally, in some embodiments of the present application, the query statement generation model includes a basic module, an adaptive module, and an integration module, and the adaptive module includes a sub-module corresponding to each preset question type information;

[0345] The generation unit can include a basic interaction subunit, an adaptive subunit, and an integration subunit, as follows:

[0346] The basic interaction subunit is configured to, by the basic module, perform feature interaction on the rewritten natural language text, the task-related field, the task-related data table, the prompt information, and the question type information to obtain feature interaction information;

[0347] The adaptive subunit is configured to, for each sub-module, perform feature interaction on the rewritten natural language text, the task-related field, the task-related data table, the prompt information, and the question type information by the sub-module to obtain sub-feature interaction information output by the sub-module;

[0348] The integration subunit is configured to, by the integration module, perform fusion processing on the feature interaction information and the sub-feature interaction information output by each sub-module to obtain the structured query statement of the rewritten natural language text.

[0349] Optionally, in some embodiments of the present application, the structured query statement generation apparatus further includes a first training unit, and the first training unit includes a first acquisition subunit, a second acquisition subunit, a first interaction subunit, a second interaction subunit, an information fusion subunit, and a first adjustment subunit, as follows:

[0350] The first obtaining subunit is configured to obtain training data, wherein the training data comprises sample natural language text and corresponding expected structured query statements;

[0351] The second obtaining subunit is configured to obtain sample associated fields, sample associated data tables, sample prompt information and sample question type information of the sample natural language text;

[0352] The first interaction subunit is configured to perform feature interaction on the sample natural language text, the sample associated fields, the sample associated data tables, the sample prompt information and the sample question type information through the basic module to obtain sample feature interaction information;

[0353] The second interaction subunit is configured to perform feature interaction on the sample natural language text, the sample associated fields, the sample associated data tables, the sample prompt information and the sample question type information through the submodules of the adaptive module to obtain sample sub-feature interaction information output by the submodules;

[0354] The information fusion subunit is configured to perform fusion processing on the sample feature interaction information and the sample sub-feature interaction information output by the submodules through the integration module to obtain an actual structured query statement of the sample natural language text;

[0355] The first adjustment subunit is configured to adjust parameters of the submodules in the adaptive module according to the actual structured query statement and the expected structured query statement to obtain a trained query statement generation model.

[0356] Optionally, in some embodiments of the present application, the query statement generation model comprises a feedforward module, an adaptive module and a normalization module; the adaptive module comprises submodules corresponding to each preset question type information, and each submodule comprises a down-sampling layer and an up-sampling layer;

[0357] The generation unit can comprise a feature interaction subunit, a down-sampling subunit, an up-sampling subunit and a normalization subunit, as follows:

[0358] The feature interaction subunit is configured to perform feature interaction on the rewritten natural language text, the task associated fields, the task associated data tables, the prompt information and the question type information through the feedforward module to obtain feature interaction information;

[0359] The down-sampling subunit is configured to perform down-sampling processing on the feature interaction information through the down-sampling layers of the submodules of the adaptive module to obtain down-sampled feature information output by the down-sampling layers of the submodules;

[0360] The up-sampling subunit is configured to perform up-sampling processing on the down-sampled feature information output by the corresponding down-sampling layers through the up-sampling layers of the submodules of the adaptive module to obtain up-sampled feature information output by the up-sampling layers of the submodules.

[0361] a normalization subunit configured to normalize the up-sampled feature information output by the up-sampling layer of each subunit through a normalization module to obtain a structured query statement of the rewritten natural language text.

[0362] Optionally, in some embodiments of the present application, the structured query statement generation apparatus further comprises a second training unit, which comprises a third acquisition subunit, a fourth acquisition subunit, a third interaction subunit, a down-sampling processing subunit, an up-sampling processing subunit, a normalization processing subunit and a second adjustment subunit, as follows:

[0363] the third acquisition subunit is configured to acquire training data, the training data comprising a sample natural language text and a corresponding expected structured query statement;

[0364] the fourth acquisition subunit is configured to acquire a sample associated field, a sample associated data table, sample prompt information and sample question type information of the sample natural language text;

[0365] the third interaction subunit is configured to perform feature interaction on the sample natural language text, the sample associated field, the sample associated data table, the sample prompt information and the sample question type information through a feedforward module to obtain sample feature interaction information;

[0366] the down-sampling processing subunit is configured to perform down-sampling processing on the sample feature interaction information through a down-sampling layer of each subunit in the adaptive module to obtain down-sampled feature information output by the down-sampling layer of each subunit;

[0367] the up-sampling processing subunit is configured to perform up-sampling processing on the down-sampled feature information output by the corresponding down-sampling layer through an up-sampling layer of each subunit in the adaptive module to obtain up-sampled feature information output by the up-sampling layer of each subunit;

[0368] the normalization processing subunit is configured to normalize the up-sampled feature information output by the up-sampling layer of each subunit through a normalization module to obtain an actual structured query statement of the sample natural language text;

[0369] the second adjustment subunit is configured to adjust parameters of each subunit in the adaptive module according to the actual structured query statement and the expected structured query statement to obtain a trained query statement generation model.

[0370] As can be seen, the embodiment can determine the natural language text representing the query task by the first determination unit 301, and determine a preset database, the preset database including at least one data table, and each data table including at least one field; determine at least one task-associated field and at least one task-associated data table associated with the natural language text from the preset database based on the natural language text by the second determination unit 302; rewrite the natural language text according to the task-associated field to obtain a rewritten natural language text by the rewriting unit 303; obtain prompt information for the rewritten natural language text by the obtaining unit 304, the prompt information being used to guide the generation of the structured query statement by the query statement generation model; perform question classification on the rewritten natural language text by the question classification unit 305 to obtain question type information of the rewritten natural language text; and generate the structured query statement of the rewritten natural language text based on the task-associated field, the task-associated data table, the prompt information and the question type information by the generation unit 306 through the query statement generation model.

[0371] The application can first determine the corresponding task-associated field and the task-associated data table based on the natural language text, rewrite the natural language text, make the natural language text more easily understood by the model through the rewriting, and finally generate the structured query statement in combination of the task-associated field, the task-associated data table, the corresponding prompt information and the question type information, so that the generation task of the query statement is more comprehensively considered, a more detailed task flow is provided, the generation ability of the model is avoided to be relied on, and the accuracy of the structured query statement generation can be greatly improved.

[0372] The embodiment of the application further provides an electronic device, as shown in FIG. 4, which shows a structural schematic diagram of the electronic device related to the embodiment of the application. The electronic device can be a terminal or a server, etc. Specifically, the electronic device can include a processor 401 having one or more processing cores, a memory 402 having one or more computer readable storage media, a power supply 403 and an input unit 404, etc.

[0373] The electronic device can include a processor 401 having one or more processing cores, a memory 402 having one or more computer readable storage media, a power supply 403 and an input unit 404, etc. Those skilled in the art can understand that the structure of the electronic device shown in FIG. 4 does not constitute a limitation on the electronic device, and can include more or fewer components than the diagram, or combine certain components, or different component arrangements. Among them:

[0374] The processor 401 is the control center of the electronic device, connects each part of the entire electronic device by various interfaces and lines, and performs various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 402 and calling data stored in the memory 402. Optionally, the processor 401 can include one or more processing cores; preferably, the processor 401 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application program, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 401.

[0375] The memory 402 can be used to store software programs and modules, and the processor 401 executes various function applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc.; and the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 402 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 402 can also include a memory controller to provide access for the processor 401 to the memory 402.

[0376] The electronic device also includes a power supply 403 for supplying power to each component, and preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 403 can also include one or more than one direct current or alternating current power supply, a recharging system, a power failure detection circuit, a power converter or inverter, a power state indicator, etc. any component.

[0377] The electronic device can also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0378] Although not shown, the electronic device can also include a display unit, etc., which will not be described here. Specifically, in the present embodiment, the processor 401 in the electronic device will load the executable file corresponding to the process of one or more than one application program into the memory 402 according to the following instructions, and run the application program stored in the memory 402 by the processor 401, so as to realize various functions, as follows:

[0379] The natural language text representing the query task is determined, and a preset database is determined, the preset database including at least one data table, each data table including at least one field; at least one task-associated field and at least one task-associated data table associated with the natural language text are determined from the preset database based on the natural language text; the natural language text is rewritten according to the task-associated field, to obtain a rewritten natural language text; prompt information for the rewritten natural language text is obtained, the prompt information being used to guide a structured query statement generation model to generate a structured query statement; the rewritten natural language text is classified to obtain problem type information of the rewritten natural language text; and the structured query statement of the rewritten natural language text is generated by the query statement generation model based on the task-associated field, the task-associated data table, the prompt information, and the problem type information.

[0380] The specific implementation of each operation can refer to the foregoing embodiments, which will not be described herein.

[0381] As can be seen, the natural language text representing the query task is determined, and a preset database is determined, the preset database including at least one data table, each data table including at least one field; at least one task-associated field and at least one task-associated data table associated with the natural language text are determined from the preset database based on the natural language text; the natural language text is rewritten according to the task-associated field, to obtain a rewritten natural language text; prompt information for the rewritten natural language text is obtained, the prompt information being used to guide a structured query statement generation model to generate a structured query statement; the rewritten natural language text is classified to obtain problem type information of the rewritten natural language text; and the structured query statement of the rewritten natural language text is generated by the query statement generation model based on the task-associated field, the task-associated data table, the prompt information, and the problem type information.

[0382] The application can first determine the corresponding task-associated field and task-associated data table based on the natural language text, and rewrite the natural language text, so that the natural language text is more easily understood by the model, and finally the structured query statement is generated in combination with the task-associated field, the task-associated data table, the corresponding prompt information, and the problem type information, which more comprehensively considers the generation task of the structured query statement, provides a more detailed task process, avoids relying only on the generation capability of the model, and can greatly improve the accuracy of the structured query statement generation.

[0383] Those skilled in the art can understand that all or part of the steps in the various methods of the foregoing embodiments can be completed by instructions, or by controlling relevant hardware by the instructions, which can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0384] To this end, an embodiment of the present application provides a computer readable storage medium, in which a plurality of instructions are stored, the instructions being loadable by a processor to perform steps in any of the structured query statement generation methods provided by the embodiments of the present application. For example, the instructions can perform the following steps:

[0385] determining natural language text representing a query task, and determining a preset database, the preset database including at least one data table, each data table including at least one field; determining, based on the natural language text, at least one task-associated field and at least one task-associated data table associated with the natural language text from the preset database; rewriting the natural language text according to the task-associated field to obtain rewritten natural language text; obtaining prompt information for the rewritten natural language text, the prompt information being used to guide a query statement generation model to generate a structured query statement; performing question classification on the rewritten natural language text to obtain question type information of the rewritten natural language text; and generating, by the query statement generation model, a structured query statement of the rewritten natural language text based on the task-associated field, the task-associated data table, the prompt information, and the question type information.

[0386] The specific implementation of each of the above operations can be referred to the foregoing embodiments, which will not be described here.

[0387] The computer readable storage medium can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0388] Since the instructions stored in the computer readable storage medium can perform steps in any of the structured query statement generation methods provided by the embodiments of the present application, the beneficial effects that can be achieved by any of the structured query statement generation methods provided by the embodiments of the present application can be achieved, which will be described in detail in the foregoing embodiments, and will not be described here.

[0389] According to an aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of an electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the electronic device to perform the method provided in any of the various optional implementations of the structured query statement generation aspect.

[0390] To sum up, the application provides a structured query statement generation method, device, equipment, computer readable storage medium and computer program product. The electronic device first determines the natural language text representing the query task and the preset database, and the preset database includes at least one data table, and each data table has at least one field. In actual business scenarios, the data of an enterprise is often stored in multiple data tables, and the number of data tables is large. Accurate determination of the preset database and the natural language text lays a foundation for subsequent query tasks. By clearly defining the data source and task description of the query, blind search in massive data is avoided, the pertinence of data query is improved, and the resource utilization is improved. Based on the natural language text, the associated task association field and the task association data table are determined from the preset database. This process is like accurately positioning the target in a complex data maze. Through analysis of the natural language text, the fields and data tables related to the query task are screened out, reducing the interference of irrelevant data, making the subsequent query operation more focused, and improving the efficiency of data processing. Rewriting the natural language text according to the task association field can make the natural language text more consistent with the understanding mode of the model. In actual application, the expression of natural language often has diversity and ambiguity, while the model is better at processing standard and explicit text. The rewritten text can reduce the difficulty of model understanding, improve the accuracy of model processing of the text, and thus improve the accuracy of the final structured query statement generation. The prompt information of the rewritten natural language text is obtained, and the prompt information is like a "navigation" for the model, guiding the model to generate a structured query statement in the correct direction. It can provide additional information and constraints for the model, reduce the search space of the model, and improve the efficiency and accuracy of the generated statement. The rewritten natural language text is classified to obtain problem type information, and different types of problems may require different processing strategies when generating a structured query statement. Through problem classification, the model can generate statements using more appropriate methods according to the characteristics and difficulty of the problem, improving the pertinence and accuracy of the generated statement. Finally, a structured query statement is generated through a query statement generation model. This comprehensive consideration of the query statement generation task provides a detailed process, avoiding relying only on the generation ability of the model, and greatly improving the accuracy of the structured query statement generation.

[0391] Further, in determining the task association field and the task association data table, the electronic device first recalls at least one candidate field and at least one candidate data table from the preset database, then calculates the similarity of each candidate field and candidate data table with the natural language text, and determines the task association field and the task association data table according to the similarity. In the data recall stage, through various recall methods (such as entity similarity recall, question similarity recall, question Elasticsearch retrieval, etc.), the possible related fields and data tables can be widely screened from the preset database. The calculation of the similarity is a further screening and optimization of the recall results, only the fields and data tables with high similarity with the natural language text are retained. This way of first extensive recall and then accurate screening not only ensures that important related data will not be missed, but also avoids processing too much irrelevant data, improving the efficiency and accuracy of data processing.

[0392] Further, in recalling the candidate field and the candidate data table, the electronic device performs entity recognition processing on the natural language text to obtain an entity set, and recalls the candidate field and the candidate data table based on the similarity of the entities in the entity set and the table information of the fields and data tables in the preset database. Entity recognition can extract key entities from the natural language text, and these entities are the core elements of the query task. By calculating the similarity of the entities and the fields and data tables in the database for recall, the elements related to the query task can be more accurately found. For example, in a data table containing a large amount of commodity information, if "mobile phone" is mentioned in the natural language text, through entity recognition and similarity calculation, the field and data table related to "mobile phone" can be quickly located, improving the accuracy and efficiency of recall.

[0393] Further, the electronic device can also determine similar questions by calculating the matching degree of the natural language text and each preset question, determine the data table associated with the similar questions as the candidate data table, and recall at least one candidate field based on the natural language text. The preset question is a summary and induction of common query questions, and by calculating the matching degree to find similar questions, existing knowledge and experience can be used to guide data recall. If there is already the best data table association for a certain type of query in the preset question, the candidate data table can be quickly determined by matching the similar question, reducing unnecessary search and calculation, and improving the efficiency of recall.

[0394] Further, the electronic device can also perform data analysis on the data tables in the preset database to determine key information, generate a retrieval index, create a retrieval index library, and find candidate data tables and candidate fields based on the natural language text in the retrieval index library. Data analysis on the data tables to determine key information can extract the core features of the data tables, which can be used as the basis for the retrieval index. Creating a retrieval index library is like establishing an efficient "catalog" for the database. Based on the natural language text, relevant data tables and fields can be quickly located in the retrieval index library, greatly improving the efficiency of data retrieval.

[0395] Further, when calculating the similarity to determine the task-related fields and task-related data tables, the electronic device filters the candidate data tables according to a preset data strategy, detects the relevance of the task-related fields and the task-related data tables through a first preset model, fuses the table identifier information and the table fields of the task-related data tables to obtain table information, fuses the table information with the entities of the natural language text through a second preset model, and predicts the similarity to determine the task-related data tables. The preset data strategy can be customized according to actual business needs, for example, to filter out some irrelevant data tables, reducing the amount of data for subsequent processing. The use of the first preset model and the second preset model further filters and analyzes the data. Through the detection and prediction of the model, the task-related fields and the task-related data tables can be more accurately determined, improving the accuracy of data filtering.

[0396] Further, before rewriting the natural language text, the electronic device filters the task-related data tables and the task-related fields through an information optimization model, detects whether the filtered task-related fields can perform a structured query task, and filters the task-related fields according to the detection result. The information optimization model is like a "data purifier" that can remove irrelevant information from the task-related data tables and the task-related fields. By detecting and filtering, only the fields related to the query task and executable are retained, reducing the interference of irrelevant information on the query statement generation model, improving the processing efficiency of the model and the accuracy of the generated structured query statement.

[0397] Further, when obtaining the prompt information, the electronic device obtains a query statement generation model, selects a target prompt library from each preset prompt library according to the rewritten natural language text, and determines the prompt information according to the correspondence between the information in the target prompt library and the rewritten natural language text. The preset prompt library contains various types of prompt information, such as time conditions, join table conditions, and calculation logic. Selecting the target prompt library according to the characteristics of the rewritten natural language text can provide more accurate guidance for the model. For example, if the rewritten natural language text contains time-related information, appropriate prompt information can be selected from the preset time condition library to help the model better handle time-related queries, improving the accuracy of generating structured query statements.

[0398] Further, when the preset prompt library includes a preset time condition library, a preset join table condition library, a preset calculation logic library, and a preset few-shot library, if the rewritten natural language text contains time entities, the preset time condition library is selected as the target prompt library, time partition condition information is retrieved to generate prompt information; if it is related to data table connection, the preset join table condition library is selected as the target prompt library, join table condition information is retrieved to generate prompt information; if it contains calculation indicators, the preset calculation logic library is selected as the target prompt library, calculation logic information is retrieved to generate prompt information; if it is related to few-shot learning problems, the preset few-shot library is selected as the target prompt library, and a structured query generation strategy is retrieved to generate prompt information. For different situations, prompt information is obtained from different preset prompt libraries, which can provide more accurate guidance for the query statement generation model. Different types of prompt information are suitable for different query scenarios, which can help the model better handle various complex query tasks and improve the accuracy and adaptability of generating structured query statements.

[0399] Further, after generating the structured query statement, the electronic device verifies the structured query statement, and if the verification is passed, queries the natural language text corresponding query result in the preset database based on the statement. Verifying the structured query statement is an important link to ensure the accuracy of data query. Through multi-level verification strategies such as syntax verification, semantic verification, and execution verification, errors that may exist in the structured query statement can be discovered and corrected in a timely manner. For example, syntax verification can check whether the statement conforms to the SQL syntax specification, and semantic verification can check whether the tables and fields used in the query statement exist in the database. Through verification, invalid queries caused by incorrect query statements are avoided, and the accuracy and efficiency of data query are improved.

[0400] Further, when classifying the rewritten natural language text, the electronic device determines its corresponding question paradigm, determines the generation difficulty level of the query statement according to the data structure association information between the task associated data tables, and determines the problem type information from each preset problem type information based on the question paradigm and the generation difficulty level. Different question paradigms and generation difficulty levels correspond to different query processing strategies. Through problem classification, the query statement generation model can generate statements using more appropriate methods according to the characteristics and difficulty of the problem. For example, for simple problems, a more concise generation strategy can be used, and for complex problems, a more complex processing method can be used, improving the relevance and accuracy of statement generation.

[0401] Further, in determining the generation difficulty level of the query statement, the electronic device performs nested table and join table detection on the task associated data table, and determines the generation difficulty level according to the detection result. Nested table and join table operations often increase the complexity of the query statement. By detecting the task associated data table, the generation difficulty of the query statement can be accurately evaluated. According to the generation difficulty level, the model can reasonably allocate computing resources, use more complex algorithms and more computing resources for queries with higher difficulty, and use simpler algorithms and fewer computing resources for queries with lower difficulty, thereby improving resource utilization and query efficiency.

[0402] Further, when the query statement generation model includes a basic module, an adaptive module, and an integration module, and the adaptive module includes sub-modules corresponding to each preset question type information, the electronic device performs feature interaction on the rewritten natural language text, the task associated field, the task associated data table, the prompt information, and the question type information through the basic module to obtain feature interaction information, each sub-module performs feature interaction to obtain sub-feature interaction information, and the integration module fuses these information to obtain a structured query statement. The basic module is responsible for preliminary processing and feature extraction of the input information, the sub-modules of the adaptive module perform special processing for different types of questions, and the integration module fuses the outputs of each module. This multi-module cooperation mode can fully utilize the advantages of different modules and process different types of questions. For example, for questions of different question paradigms and generation difficulty levels, the sub-modules can learn specific features and patterns, and through the fusion of the integration module, the accuracy and adaptability of generating structured query statements are improved.

[0403] Further, before performing feature interaction through the basic module, the electronic device obtains training data including sample natural language text and expected structured query statement, obtains associated fields, associated data tables, prompt information, and question type information of the sample, performs feature interaction through the basic module and the sub-modules, fuses the information to obtain an actual structured query statement, adjusts the parameters of each sub-module in the adaptive module according to the actual and expected structured query statements to obtain a trained query statement generation model. By training the model with training data, the model can learn the features and patterns of different types of questions. Adjusting the parameters of the sub-modules according to the differences between the actual and expected structured query statements can continuously optimize the performance of the model. For example, for a certain type of question, the actual structured query statement generated by the model differs from the expected structured query statement. By adjusting the parameters of the sub-modules, the model can better handle this type of question, thereby improving the accuracy of generating structured query statements.

[0404] Further, when the query statement generation model comprises a feedforward module, an adaptive module, and a normalization module, the adaptive module comprises a sub-module corresponding to each preset question type information, each sub-module has a down-sampling layer and an up-sampling layer, the electronic device obtains feature interaction information through feature interaction of the feedforward module, the down-sampling layer and the up-sampling layer of the sub-module process to obtain feature information after up-sampling, and the normalization module performs normalization processing to obtain a structured query statement. The feedforward module performs feature interaction on input information to extract key features. The down-sampling layer and the up-sampling layer of the sub-module perform dimension reduction and dimension increase processing on the features, which can reduce feature redundancy and improve feature expression capability. The normalization module performs normalization processing on the features, so that the features are compared and fused under a unified scale, thereby improving the accuracy of generating a structured query statement.

[0405] Further, before feature interaction through the feedforward module, the electronic device obtains training data, obtains related information of samples, processes through the feedforward module and the down-sampling layer and the up-sampling layer of the sub-module, and obtains an actual structured query statement through normalization of the normalization module. The parameters of each sub-module in the adaptive module are adjusted according to the actual and expected structured query statements to obtain a trained query statement generation model. The model is trained through the training data, so that the model learns the features and patterns of different types of problems. The parameters of the sub-module are adjusted according to the difference between the actual and expected structured query statements, so that the performance of the model can be continuously optimized. For example, for a certain type of problem, there is a difference between the actual structured query statement generated by the model and the expected structured query statement. By adjusting the parameters of the sub-module, the model can better handle this type of problem, thereby improving the accuracy of generating a structured query statement.

[0406] In the information recall stage, information recall can be performed in a collaborative filtering manner, including user-user collaborative filtering, item-item collaborative filtering, and matrix decomposition.

[0407] User-user collaborative filtering: first, a user vector is constructed according to the historical operation records of the user on the natural language text, then the similarity between different users is calculated to find a similar user group to the current user. Analyze the fields and data tables frequently queried by the similar user group in the preset database, and take these fields and data tables as candidate objects for recall. In actual application, different users may have similar query needs. By finding a similar user group, the query experience of other users can be used to guide the query of the current user. For example, in an e-commerce data query scenario, if multiple similar users frequently query the price information of a certain type of goods, then for the current user, the related price field and goods data table can also be taken as candidate objects for recall, thereby improving the accuracy and efficiency of recall.

[0408] Item-item collaborative filtering: Calculate the similarity between each field and data table in the pre-set database. When the natural language text is related to a certain field or data table, recall other fields and data tables with high similarity. In the context of close data association, the relevant information of a field or data table often has associations with other fields or data tables. Through item-item collaborative filtering, these potential associations can be mined. For example, in a data table containing product information and sales information, if the natural language text is related to the product name, item-item collaborative filtering can recall the sales data fields and data tables related to the product, improving the comprehensiveness of recall.

[0409] Matrix decomposition: such as singular value decomposition (SVD), probability mass function (PMF), etc., used to discover the hidden features of users and items. Matrix decomposition can decompose high-dimensional data matrices into low-dimensional matrices, thereby discovering the hidden features in the data. In data recall, matrix decomposition can find the potential associations between users and fields, data tables. For example, in a data matrix containing user action information and product information, matrix decomposition can discover the user's preference for certain product features, thereby more accurately recalling related fields and data tables, improving the accuracy of recall.

[0410] When rewriting natural language text according to task-related fields, a question rewriting model based on reinforcement learning can be used. The training data of this model includes a large number of natural language texts, corresponding task-related fields, and artificially rewritten texts. In the training process, a policy gradient algorithm is used, with the similarity between the rewritten text and the target text as the reward signal, to continuously optimize the rewriting strategy of the model. At the same time, an adversarial training mechanism is introduced to train a discriminator to distinguish between artificially rewritten texts and model-generated texts, prompting the generator to generate texts closer to artificially rewritten texts.

[0411] Policy gradient algorithm optimizes rewriting strategy: The policy gradient algorithm continuously adjusts the parameters of the model so that the similarity between the rewritten text generated by the model and the target text becomes higher and higher. In practical applications, the rewriting of natural language text needs to consider the understanding ability of the model and the final query effect. Through the policy gradient algorithm, the model can learn a better rewriting strategy, improving the quality of rewriting. For example, in the rewriting of a natural language text containing professional terms, the model can learn to find a more appropriate expression, making the rewritten text easier for the model to understand, thereby improving the accuracy of the final structured query statement generation.

[0412] The adversarial training mechanism improves the quality of rewriting: In the adversarial training mechanism, the discriminator and the generator are in a game with each other. The role of the discriminator is to distinguish whether the rewritten text is a real human rewriting or a model generation, and the generator adjusts its parameters according to the feedback of the discriminator. This mechanism prompts the generator to generate text closer to human rewriting. In practical applications, human rewriting text often has higher quality and accuracy. Through adversarial training, the rewritten text generated by the model can be closer to the level of human rewriting, improving the quality of rewriting and the accuracy of the generated structured query statement.

[0413] In the structured query statement generation and optimization execution layer, a fine-tuning scheme of LORA (Low-Rank Adaptation) + MOE (Mixture of Experts) is designed, and a LORA module is trained for data of different problem classifications, and the MOE algorithm is used for combination in the high layer.

[0414] Specific learning of LORA module: LORA module is a lightweight and efficient model adaptation method, which realizes the rapid adaptation to specific tasks by introducing low-rank matrices in the key parts of the model. For each problem classification, an independent LORA module is set, which focuses on learning the features and patterns related to the classification. In practical applications, different types of problems may have different features and patterns. By setting an independent LORA module for each problem classification, the model can learn the features of different types of problems more specifically, improving the model's processing ability for different problems. For example, for the maximum value problem and the proportion problem, different LORA modules are used for learning, which can better capture the characteristics of these two types of problems and improve the accuracy of generating structured query statements.

[0415] Model integration of MOE algorithm: MOE algorithm is an effective model integration technology, which improves the overall performance by weighting and combining the outputs of multiple expert models. Each LORA module is regarded as an expert, and the MOE algorithm is used to combine them in the high layer to realize multi-task through multi-LORA+MOE, so as to realize accurate identification of different problem classifications. In practical applications, different LORA modules may have different advantages in different types of problems. By weighting and combining the outputs of these modules through the MOE algorithm, the advantages of each module can be fully utilized, improving the overall performance of the model. For example, when dealing with multiple types of query problems, the MOE algorithm can reasonably allocate the weight of each LORA module according to the type and difficulty of the problem, so as to more accurately generate structured query statements and improve the accuracy of generation.

[0416] Any combination of the technical features in the above embodiments can be made, and for the sake of brevity, not all possible combinations are described above, however, as long as the combination of the technical features does not exist in contradiction, it shall be considered within the scope of the present disclosure.

[0417] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it shall not be understood as a limitation on the patent scope of the present application. It shall be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these shall be within the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A structured query statement generation method, executed by an electronic device, comprising: determining a natural language text representing a query task, and determining a preset database, the preset database comprising at least one data table, each data table comprising at least one field; determining, based on the natural language text, at least one task-associated field and at least one task-associated data table associated with the natural language text from the preset database; rewriting the natural language text according to the task-associated field to obtain a rewritten natural language text; obtaining prompt information for the rewritten natural language text, the prompt information being used to guide a query statement generation model to generate a structured query statement; performing question classification on the rewritten natural language text to obtain question type information of the rewritten natural language text; and generating, by the query statement generation model, a structured query statement of the rewritten natural language text based on the task-associated field, the task-associated data table, the prompt information, and the question type information.

2. The method of claim 1, wherein the determining, based on the natural language text, at least one task-associated field and at least one task-associated data table associated with the natural language text from the preset database comprises: recalling, based on the natural language text, at least one candidate field and at least one candidate data table from the preset database; calculating a similarity of each candidate field and each candidate data table with the natural language text, to determine a task-associated field from the candidate fields and a task-associated data table from the candidate data tables according to the similarity.

3. The method of claim 2, wherein the recalling, based on the natural language text, at least one candidate field and at least one candidate data table from the preset database comprises: performing entity recognition processing on the natural language text to obtain an entity set corresponding to the natural language text; recalling, based on a similarity between entities in the entity set and fields in the preset database, at least one candidate field from the preset database; recalling, based on a similarity between entities in the entity set and table information of data tables in the preset database, at least one candidate data table from the preset database.

4. The method of claim 2, wherein the recalling, based on the natural language text, at least one candidate field and at least one candidate data table from the preset database comprises: calculating a matching degree between the natural language text and each preset question, each preset question being associated with at least one data table in the preset database; determining, according to the matching degree, a similar question from the preset questions; determining, as a candidate data table, a data table associated with the similar question; recalling, based on the natural language text, at least one candidate field from the preset database.

5. The method of claim 2, wherein the recalling, based on the natural language text, at least one candidate field and at least one candidate data table from the preset database comprises: ​ perform data analysis on a data table in a preset database to determine key information of the data table; generate a search index of the data table according to the key information of the data table; create a search index library based on the search index, the search index library including a mapping relationship between the search index and the data table; perform data table and field searching in the search index library based on a natural language text to obtain at least one candidate data table and candidate field.

6. The method of any one of claims 2 to 5, wherein the calculating of the similarity of each candidate field and each candidate data table with the natural language text respectively, and the determining of a task-related field from the candidate fields and a task-related data table from the candidate data tables according to the similarity, comprises: filtering the candidate data tables according to a preset data strategy to obtain filtered candidate data tables; detecting the relevance of each candidate field and each filtered candidate data table with the natural language text through a first preset model to determine a task-related field from each candidate field and a related data table from each filtered candidate data table; fusing table identifier information and table fields of the related data table to obtain table information; fusing the table information and entities of the natural language text through a second preset model to obtain fused information; extracting feature information of the fused information to predict the similarity of the related data table and the natural language text based on the feature information; and determining a task-related data table from the related data tables according to the similarity.

7. The method of any one of claims 1 to 6, wherein the rewriting of the natural language text according to the task-related field to obtain a rewritten natural language text further comprises: screening a task-related data table and a task-related field based on the natural language text through an information optimization model to obtain a screened task-related data table and a screened task-related field; detecting whether the screened task-related field can perform a structured query task of the natural language text based on the screened task-related data table; and filtering the screened task-related field based on a detection result to obtain a filtered task-related field.

8. The method of any one of claims 1 to 7, wherein the obtaining of prompt information for the rewritten natural language text comprises: obtaining a query statement generation model; selecting a target prompt library from each preset prompt library according to the rewritten natural language text, and determining the prompt information of the rewritten natural language text according to a corresponding relationship between information in the target prompt library and the rewritten natural language text.

9. The method of claim 8, wherein the preset prompt library comprises a preset time condition library, a preset join table condition library, a preset calculation logic library, and a preset few-shot library. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ The method according to any one of claims 1 to 9, after the structured query statement of the rewritten natural language text is generated based on the task-associated field, the task-associated data table, the prompt information and the question type information by the query statement generation model, the method further comprises: verifying the structured query statement; when the verification result is passed, querying in the preset database based on the structured query statement to obtain a query result corresponding to the natural language text.

11. The method according to claim 1, wherein the question classification of the rewritten natural language text to obtain the question type information of the rewritten natural language text comprises: determining a question paradigm corresponding to the rewritten natural language text; determining a generation difficulty level of a query statement corresponding to the task-associated data table according to data structure association information between the task-associated data tables; determining the question type information of the rewritten natural language text from each preset question type information based on the question paradigm and the generation difficulty level, wherein each preset question type information corresponds to a preset question paradigm and a preset generation difficulty level.

12. The method according to claim 11, wherein the determination of the generation difficulty level of the query statement corresponding to the task-associated data table according to the data structure association information between the task-associated data tables comprises: detecting nesting and table connection of the task-associated data table; determining the generation difficulty level of the query statement corresponding to the task-associated data table according to the detection result. ​ ​ ​ ​ ​ 13. The method of any one of claims 1-9, wherein the query statement generation model comprises a base module, an adaptive module, and an integration module, and the adaptive module comprises a plurality of sub-modules corresponding to a plurality of preset question type information; generating, by the query statement generation model, the structured query statement of the rewritten natural language text based on the task-related field, the task-related data table, the prompt information, and the question type information, comprises: performing, by the base module, feature interaction on the rewritten natural language text, the task-related field, the task-related data table, the prompt information, and the question type information to obtain feature interaction information; for each sub-module, performing, by the sub-module, feature interaction on the rewritten natural language text, the task-related field, the task-related data table, the prompt information, and the question type information to obtain sub-feature interaction information output by the sub-module; performing, by the integration module, fusion processing on the feature interaction information and the sub-feature interaction information output by each sub-module to obtain the structured query statement of the rewritten natural language text.

14. The method of claim 13, wherein before performing, by the base module, feature interaction on the rewritten natural language text, the task-related field, the task-related data table, the prompt information, and the question type information to obtain feature interaction information, the method comprises: obtaining training data comprising a sample natural language text and an expected structured query statement corresponding to the sample natural language text; obtaining a sample-related field, a sample-related data table, a sample prompt information, and a sample question type information of the sample natural language text; performing, by the base module, feature interaction on the sample natural language text, the sample-related field, the sample-related data table, the sample prompt information, and the sample question type information to obtain sample feature interaction information; for each sub-module of the adaptive module, performing, by the sub-module, feature interaction on the sample natural language text, the sample-related field, the sample-related data table, the sample prompt information, and the sample question type information to obtain sample sub-feature interaction information output by the sub-module; performing, by the integration module, fusion processing on the sample feature interaction information and the sample sub-feature interaction information output by each sub-module to obtain an actual structured query statement of the sample natural language text; adjusting parameters of each sub-module in the adaptive module according to the actual structured query statement and the expected structured query statement to obtain a trained query statement generation model.

15. The method of any one of claims 1-9, wherein the query statement generation model comprises a feed-forward module, an adaptive module, and a normalization module, and the adaptive module comprises a plurality of sub-modules corresponding to a plurality of preset question type information, and each sub-module comprises a down-sampling layer and an up-sampling layer. The generating model generates the structured query statement of the rewritten natural language text based on the task association field, the task association data table, the prompt information and the question type information. The feature interaction information is obtained by the feedforward module through feature interaction of the rewritten natural language text, the task association field, the task association data table, the prompt information and the question type information. The feature information after down-sampling is obtained by the down-sampling layer of each sub-module in the adaptive module through down-sampling processing of the feature interaction information. The feature information after up-sampling is obtained by the up-sampling layer of each sub-module in the adaptive module through up-sampling processing of the feature information after down-sampling output by the corresponding down-sampling layer. The structured query statement of the rewritten natural language text is obtained by the normalization module through normalization processing of the feature information after up-sampling output by the up-sampling layer of each sub-module.

16. The method of claim 15, before the feature interaction information is obtained by the feedforward module through feature interaction of the rewritten natural language text, the task association field, the task association data table, the prompt information and the question type information, further comprising: obtaining training data, the training data comprising sample natural language text and its corresponding expected structured query statement; obtaining sample association field, sample association data table, sample prompt information and sample question type information of the sample natural language text; obtaining sample feature interaction information by the feedforward module through feature interaction of the sample natural language text, the sample association field, sample association data table, sample prompt information and sample question type information; obtaining feature information after down-sampling output by the down-sampling layer of each sub-module in the adaptive module through down-sampling processing of the sample feature interaction information; obtaining feature information after up-sampling output by the up-sampling layer of each sub-module in the adaptive module through up-sampling processing of the feature information after down-sampling output by the corresponding down-sampling layer; obtaining actual structured query statement of the sample natural language text by the normalization module through normalization processing of the feature information after up-sampling output by the up-sampling layer of each sub-module; adjusting parameters of each sub-module in the adaptive module according to the actual structured query statement and the expected structured query statement to obtain the trained query statement generation model.

17. A structured query statement generation device, comprising: a first determination unit configured to determine a natural language text representing a query task and determine a preset database, the preset database comprising at least one data table, and each data table comprising at least one field; a second determination unit configured to determine at least one task association field and at least one task association data table associated with the natural language text from the preset database based on the natural language text. The rewriting unit is configured to rewrite the natural language text according to the task association field, to obtain rewritten natural language text. The obtaining unit is configured to obtain prompt information for the rewritten natural language text, the prompt information being used to guide a query statement generation model to generate a structured query statement. The question classification unit is configured to perform question classification on the rewritten natural language text, to obtain question type information of the rewritten natural language text. The generating unit is configured to generate, by the query statement generation model, a structured query statement of the rewritten natural language text based on the task association field, the task association data table, the prompt information, and the question type information. 18.An electronic device, comprising a memory and a processor, wherein the memory stores an application, and the processor is configured to run the application in the memory to perform operations in the structured query statement generation method according to any one of claims 1 to 16. 19.A computer readable storage medium, storing a plurality of instructions, wherein the instructions are adapted to be loaded by a processor to perform steps in the structured query statement generation method according to any one of claims 1 to 16. 20.A computer program product, comprising a computer program or instructions, wherein the computer program or instructions are executed by a processor to implement steps in the structured query statement generation method according to any one of claims 1 to 16. ​

Images