NL2SQL method and device based on multi-view feature decoupling

Abstract

The present invention discloses a NL2SQL method and device based on multi-view feature decoupling, the method specifically comprising the following steps: (1) NL2SQL knowledge base construction step; (2) NL2SQL data set construction step; (3) NL2SQL model construction step, for respectively constructing the semantic encoding model of the SELECT part, the WHERE part and the SQL query statement as a whole; in order to effectively capture the semantic features of each part of SELECT and WHERE, a method for multi-view feature decoupling is constructed; (4) NL2SQL model training step, for constructing a loss function and an optimization function. The method of the present invention has achieved remarkable results on a real NL2SQL data set, and has great application prospects in the field of automatic question answering systems.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and natural language processing technology, and in particular to an NL2SQL method and device based on multi-view feature decoupling. Background Art

[0002] With the development of informatization and information technology, a large amount of business data is generated every moment. If users want to query data, they usually use the fixed query function on the relevant system page, which requires users to be very familiar with the system functions in order to quickly locate specific modules. At the same time, the query conditions and return content on the system page have a fixed format. If you want to meet personalized queries, you need to customize new page functions, which will incur additional costs.

[0003] At present, these large amounts of data are generally organized and stored using databases, among which relational databases are the mainstream of database applications. They use concepts and methods such as set algebra to process data in the database, and have the advantages of being easy to understand, easy to use, easy to maintain, and supporting SQL (Structured Query Language). Although general queries can be obtained in the database through SQL, SQL, as a database operation language, is essentially a programming language. It requires operators to be trained in database and SQL related knowledge and have certain professional knowledge before they can be more proficient in SQL statement programming. In addition to having relevant knowledge of SQL and database technology, users also need to have some understanding of the model information of the relational database involved in actual operation, so that various operation requirements can be converted into correct SQL statements. At the same time, with the increasing number of application scenarios of database systems, the amount of data and internal logic of the database are becoming more and more complex. Even for experienced software developers or database managers, it is becoming more and more difficult to convert their query intentions into correct SQL statements.

[0004] With the rapid development of artificial intelligence technology in recent years, semantic recognition technology has become increasingly mature, providing the prerequisite for the application of NL2SQL (Natural Language to SQL, which converts natural language into SQL language) technology in information systems. NL2SQL solves the problem of freely querying databases through natural language, and solves the problem that users who are not familiar with the system can also query data on demand in natural language at the front desk, reducing the requirements for business personnel and lowering the distance and threshold of human-computer interaction. Summary of the invention

[0005] Purpose of the invention: How to accurately convert user questions into SQL query statements and return the answers required by users is a core issue of NL2SQL. This patent application solves this problem with the help of deep learning technology. In order to solve the semantic loss problem that occurs during the encoding process, this application proposes a new multi-stage semantic encoding model, which is used to perform semantic extraction of the SELECT part, the WHERE part of the SQL query statement and the SQL as a whole; in order to effectively capture the semantic features of each part of SELECT and WHERE, this application proposes an NL2SQL method and device based on multi-perspective feature decoupling. The method proposed in this application has achieved remarkable results on a real NL2SQL dataset and has great application prospects in the field of automatic question answering systems.

[0006] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0007] A NL2SQL method based on multi-view feature decoupling, the method comprising the following steps:

[0008] S1. Build the NL2SQL knowledge base, use the user's query questions, manually write SQL query statements, construct data entries that correspond to query questions, SQL query statements, and table structures, and form the NL2SQL knowledge base;

[0009] S2. Construct the NL2SQL dataset. For each question, there is a matching SQL query statement and database table structure in the NL2SQL knowledge base. The training set, validation set, and test set are divided according to the NL2SQL knowledge base.

[0010] S3. Build the NL2SQL model, sequentially build the input layer, build the SELECT part encoding model, build the WHERE part encoding model, build the SELECT+WHERE part encoding model, build the multi-view decoupling layer for different tasks, and build the prediction layer for different subtasks;

[0011] S4. NL2SQL model training: Use the training set obtained in step S2 to train the NL2SQL model constructed in step S3, so as to construct a loss function and an optimization function.

[0012] Furthermore, the construction of the input layer in step S3 is specifically as follows:

[0013] The input layer obtains user questions and table structures from the NL2SQL dataset and formalizes them as: Input; the input layer uses BERT input, which consists of three parts: token embedding, segment embedding, and position embedding. The input vector of each word in the sentence can be obtained by adding these three vectors. The input vector of each word in the sentence is combined into a matrix, which is the input of the sentence; BERT input is subjected to several multi-head attention mechanisms, layer normalization, and feedforward network layers to obtain the output:

[0014] E=BERT(Input)

[0015] Define the classification probability distribution P for the matrix E c The calculation formula is:

[0016] P c (E) = Softmax(Wtanh(E))

[0017]

[0018] W represents a trainable parameter. Tanh is a common activation function in machine learning. The softmax function is used to normalize all values ​​in the vector and convert them into probability values. The sum of the normalized values ​​is 1.

[0019] Furthermore, the construction of the SELECT part encoding model in step S3 is specifically as follows:

[0020] The optimization goal of the SELECT encoding model only includes the subtasks SELECT-COL and SELECT-AGG (SELECT COUNT(*)) of the SELECT part (SELECT part in SQL), which are encoded using BERT to the corresponding text representation E S ;

[0021] According to E S The predicted column name is SELECT-COL, and the value range is all column names in the current data table. is the weight of the SELECT-COL prediction module, and the classification probability distribution of the SELECT-COL prediction module The formula is as follows:

[0022]

[0023] After selecting the column name COL to be used, you also need to predict the aggregation operation SELECT-AGG corresponding to the column name. The value range is {"NONE", "MAX", "MIN", "COUNT", "SUM", "AVG"}. is the weight of the SELECT-AGG prediction module:

[0024]

[0025] in, Probability distribution of the SELECT-AGG prediction module.

[0026] Furthermore, the construction of the WHERE part coding model in step S3 is specifically as follows:

[0027] The optimization goal of the WHERE encoding model only includes the subtasks WHERE-COL, WHERE-NUM, WHERE-OP, and WHERE-VALUE of the WHERE part, which are encoded using BERT to the corresponding text representation E W , according to E W Predict the WHERE subtask. First, predict the number of WHERE-NUM used by the column name COL. is the weight of the WHERE-NUM prediction module, and the formula is as follows:

[0028]

[0029] in, The classification probability distribution of the WHERE-NUM prediction module;

[0030] After predicting the number of column names used, you need to predict which column names are used for the corresponding number, that is, you need to select the column name COL to be used, and the value range is all column names in the current data table. is the weight of the WHERE-COL prediction module, and the classification probability distribution of the WHERE-COL prediction module The formula is as follows:

[0031]

[0032] After selecting the column name COL to be used, it is also necessary to predict the comparison operation WHERE-OP corresponding to the column name. The value range is {">", "<", "=", "!=", ">=", "<="}. Assume that the currently selected column name COL is the hth column in the data table T. i List, is the weight of the WHERE-OP prediction module, and the classification probability distribution of the WHERE-OP prediction module The formula is as follows:

[0033]

[0034] By performing sequence labeling on the original natural language query, the labeling contains <start>and <end>Two special tags, the text between the two tags is the obtained conditional value; using this overall feature expression, a classification model is constructed to determine whether the candidate condition constituted by the VAL value of the current text type is in the SQL statement. The value range of the binary classification problem is {0, 1}. as well as are WHERE-VAL start Prediction Module and WHERE-VAL end The weight of the prediction module,

[0035]

[0036]

[0037] in, For WHERE-VAL start The probability distribution of the prediction module, For WHERE-VAL end Probability distribution of prediction modules.

[0038] Furthermore, the coding model for constructing the SELECT+WHERE part is specifically as follows:

[0039] The optimization goal of the SELECT+WHERE encoding model includes the subtasks SELECT-COL and SELECT-AGG of the SELECT part, and the subtasks WHERE-COL, WHERE-NUM, WHERE-OP and WHERE-VALUE of the WHERE part, which are encoded using BERT to the corresponding text representation E SW , and make classification predictions for each subtask.

[0040] Furthermore, constructing a multi-view decoupling layer for different tasks is as follows:

[0041] Based on the text representation E obtained for the SELECT part S , the text representation of the WHERE part W , and the text representation of the SELECT part and the WHERE part SW On this basis, for E S and E W Use E SW Perform orthogonal projection and get E W1 and E S1 :

[0042] First, E S Projection to E SW Get E in the direction S* , the specific method is formulated as: E S* =Proj(E S ,E SW ),in, x and y are the corresponding eigenvectors;

[0043] Ask E again S With E S -E S* Projection

[0044]

[0045] here It represents the purified vector E after eliminating some SELECT features. W1 ; Similarly, we get E S1 ;

[0046] Late Integration S and E S1 Get E S2 Perform predictions on some of the SELECT subtasks and integrate E W and E W1 Get E W2 Perform prediction of some subtasks in WHERE, that is

[0047] E S2 =E S +E S1

[0048] E W2 =E W +E W1

[0049] In this way, the semantic feature vectors of the SELECT part and the WHERE part are obtained from multiple perspectives.

[0050] Furthermore, constructing different subtask prediction layers is as follows:

[0051] The semantic feature vector E of the SELECT part and the WHERE part is obtained S2 and E W2 Used for prediction of each subtask.

[0052] Furthermore, the training of the NL2SQL model is as follows:

[0053] Construct the loss function:

[0054] Using the cross entropy loss function, the formula for the cross entropy loss function loss(P(x),y) of a classification model is as follows:

[0055]

[0056] Where N represents the number of categories, y j represents the label value of the jth class, which can be 0 or 1. j ) represents the probability value of the j-th category prediction, and the overall objective function L of the model is the sum of all subtask loss functions;

[0057] Constructing the optimization function:

[0058] The deep learning model used is based on the pre-trained model BERT plus task-specific classification layer fine-tuning. The specific classification layer fine-tuning is performed during the fine-tuning process, and the number of iterations is set to 10. Each deep learning model is implemented based on the deep learning framework Pytorch. During the training process, the learning rate size set is 3e-5, the batch size is 128, the optimizer is AdamW, and the regularization method uses Dropout to prevent overfitting.

[0059] The present invention also provides a NL2SQL implementation device based on multi-view feature decoupling, the device comprising:

[0060] The NL2SQL knowledge base construction unit is used to use the user's query questions, manually write SQL query statements, construct data entries that correspond to query questions, SQL query statements and table structures, and preliminarily build a knowledge base;

[0061] The NL2SQL data set construction unit is used to divide the training set, the validation set and the test set according to the NL2SQL knowledge base to construct the NL2SQL data set;

[0062] NL2SQL model building unit, used to build the semantic coding model of the SELECT part, the WHERE part and the SQL query statement as a whole;

[0063] NL2SQL model training unit, used to build loss function and optimization function, including:

[0064] Loss function building unit, the objective function of the model is the sum of all subtask loss functions, and the model is trained to minimize this sum of objective functions;

[0065] The optimization function building unit is responsible for optimizing model parameters and adjusting the update of parameters in the model.

[0066] Furthermore, the NL2SQL model building unit also includes:

[0067] The SELECT part encodes the model building unit. The optimization target of this unit only includes the subtasks SELECT-COL and SELECT-AGG of the SELECT part, which are encoded using BERT to the corresponding text representation E S , according to E S Predict the column name SELECT-COL and the aggregation operation SELECT-AGG;

[0068] The WHERE part encodes the model building unit. The optimization target of this unit only includes the subtasks WHERE-COL, WHERE-NUM, WHERE-OP and WHERE-VALUE of the WHERE part, which are encoded using BERT to the corresponding text representation E W , according to E W Predict the corresponding subtasks of the WHERE part;

[0069] The SELECT+WHERE part encoding model building unit, the optimization target of which includes the subtasks SELECT-COL and SELECT-AGG of the SELECT part, and the subtasks WHERE-COL, WHERE-NUM, WHERE-OP and WHERE-VALUE of the WHERE part, which are encoded using BERT to the corresponding text representation E SW ; In addition, the same prediction method as each subtask in the WHERE part coding model construction unit and the SELECT part coding model construction unit is adopted to make predictions;

[0070] The multi-view decoupling construction unit for different tasks obtains the text feature vector E for the SELECT part and the WHERE part by the SELECT part encoding model construction unit and the WHERE part encoding model construction unit. S and E W ; In addition, after the SELECT+WHERE part encoding model construction unit, the text feature vector E for these two parts is obtained SW On this basis, for E S and E W Use E SW Perform orthogonal projection and get E W1 and E S1 ; E obtained by later integration S and E S1 Perform predictions on some of the SELECT subtasks and integrate the obtained E W and E W1 Perform prediction on some subtasks in WHERE;

[0071] Different subtask prediction building units are used to use the feature vectors of the SELECT part and the WHERE part obtained in the different task multi-view decoupling building units for the prediction of each subtask. The prediction of the subtask will follow the methods in the SELECT part encoding model building unit and the WHERE part encoding model building unit.

[0072] Beneficial effects:

[0073] 1. The present invention adopts a stage training method to effectively model natural language questions and table structures and capture corresponding semantic features.

[0074] 2. Through parameter sharing, the correlation constraints between subtasks in the SELECT and WHERE tasks can be implemented respectively.

[0075] 3. The present invention proposes to integrate multi-perspective semantic features, which can effectively avoid the phenomenon of semantic loss.

[0076] 4. The present invention proposes a feature decoupling method that can better capture the specificity between the two tasks of SELECT and WHERE.

[0077] 5. The present invention defines and implements a complete set of NL2SQL methods and devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is a flowchart of a NL2SQL method based on multi-view feature decoupling;

[0079] Figure 2 Flowchart for building the NL2SQL knowledge base;

[0080] Figure 3 Flowchart for constructing the NL2SQL task dataset;

[0081] Figure 4 Flowchart for building the NL2SQL model;

[0082] Figure 5 Flowchart for training the NL2SQL model;

[0083] Figure 6 It is a structural diagram of a NL2SQL device based on multi-view feature decoupling;

[0084] Figure 7 A schematic diagram of the structure of the NL2SQL model based on multi-view feature decoupling;

[0085] Figure 8 It is a schematic diagram of the original NL2SQL method based on the slot filling prediction model. DETAILED DESCRIPTION

[0086] This application proposes a NL2SQL method and device based on multi-view feature decoupling. The method proposed in this application has achieved remarkable results on a real NL2SQL dataset and has great application prospects in the field of automatic question answering systems.

[0087] Converting natural language into SQL statements has the following meanings:

[0088] 1. Assist business personnel to conveniently query data: Natural language processing, artificial intelligence and other technologies can be used to automatically analyze business personnel's problems, assist analysts to quickly obtain the required data, so that users do not need to be proficient in using the system or SQL language, and can focus more energy and time on data analysis.

[0089] 2. Improve the scalability of system query functions: Since business systems generally have fixed functions, both query conditions and the format of returned results are fixed, and their own limitations make it impossible for users to perform adaptive data queries. Using relational databases and NL2SQL technology, users only need to ask questions without additional customization of functions to obtain data results.

[0090] Terminology explanation:

[0091] The premise of feature decoupling is feature learning. Decoupling is to separate task-related features from irrelevant features. Taking classification as an example, what is decoupled are category features and irrelevant features such as background and style.

[0092] BERT: BERT stands for Bidirectional Encoder Representation from Transformers, which is a pre-trained language representation model. It emphasizes that it no longer uses the traditional unidirectional language model or the shallow concatenation of two unidirectional language models for pre-training as in the past, but uses a new masked language model (MLM) to generate deep bidirectional language representation.

[0093] Each implementation step of the method of the present application is as follows Figure 1 As shown, the specific steps are as follows:

[0094] Step 1: Build the NL2SQL knowledge base.

[0095] Using the user's query questions, manually construct SQL query statements and initially build a knowledge base, such as Figure 2 shown.

[0096] For single-sentence query business scenarios, each query statement corresponds to a data table, which contains all the column names of the table and the corresponding data records. In the annotated data, the columns selected by SQL, the corresponding aggregate functions of the columns, the relationship between conditions, the condition columns, the condition types, the condition values, etc. need to be extracted from the query statement. The SQL statement generated by combining the annotated data can be executed on the corresponding data table and can return real and valid results.

[0097] The annotation examples are shown below.

[0098]

[0099] Step 2: Build the NL2SQL dataset.

[0100] For each question, there is a matching SQL query statement and database table structure in the knowledge base. Figure 3 As shown, the training data set, validation set, and test set can be set according to the size of the collected NL2SQL knowledge base.

[0101] Step 3: Build the NL2SQL model.

[0102] The original NL2SQL approach is based on a slot filling prediction model, such as Figure 8 shown.

[0103] In the natural language description, there is a clear boundary between the SELECT part and the WHERE part of the query statement, for example, count the number of Qinghai Province's returned goods (corresponding to the WHERE part (WHERE OPERATE_STATE = 'return' AND REGION = 'Qinghai Province')) / the number of substations (corresponding to the SELECT part (SELECT COUNT(*))). The present invention proposes a corresponding improvement method for the problems existing in the current task, such as Figure 4 Step 3 sequentially constructs the input layer, the SELECT part encoding model, the WHERE part encoding model, the SELECT+WHERE part encoding model, the different task multi-view decoupling layer and the different sub-task prediction layer.

[0104] 3.1 Constructing the input layer

[0105] The input layer obtains user questions and table structures from NL2SQL respectively, and formalizes them as: Input: (Question, Schema[i]), for example: ("Count the number of substations that have been decommissioned in Qinghai Province", "Asset unit", "Dispatching agency", "Decommissioning date", "Substation ID", "Substation name", "Commissioning date", "Operation status", "Grid", "Administrative division", "AC maximum voltage level", "Type")

[0106] BERT is the basis of the NL2SQL model proposed in this paper. The input of BERT consists of three parts: token embedding, segment embedding, and position embedding. The input vector of each word in the sentence can be obtained by adding these three vectors. The input vector of each word in the sentence is combined into a matrix, which is the input of the sentence. The input of BERT is output after several multi-head attention mechanisms, layer normalization, and feed-forward network layers.

[0107] E=BERT(Input)

[0108] Define the classification probability distribution P for the matrix E c The calculation formula is:

[0109] P c (E) = Softmax(Wtanh(E))

[0110]

[0111] W represents a trainable parameter. Tanh is a common activation function in machine learning. The softmax function is used to normalize all values ​​in the vector and convert them into probability values. The sum of the normalized values ​​is 1.

[0112] 3.2 Constructing the SELECT Partial Encoding Model

[0113] The optimization goal of the SELECT encoding model only includes the subtasks SELECT-COL and SELECT-AGG (SELECT COUNT(*)) of the SELECT part (SELECT part in SQL), which are encoded using BERT to the corresponding text representation E S . ;

[0114] According to E S Predict the column name SELECT-COL, the value range is all column names in the current data table is the weight of the SELECT-COL prediction module, and the classification probability distribution of the SELECT-COL prediction module The formula is as follows:

[0115]

[0116] After selecting the column name COL to be used, you also need to predict the aggregation operation SELECT-AGG corresponding to the column name. The value range is {"NONE", "MAX", "MIN", "COUNT", "SUM", "AVG"}. is the weight of the SELECT-AGG prediction module:

[0117]

[0118] in, Probability distribution of the SELECT-AGG prediction module.

[0119] 3.3 Constructing the WHERE part coding model

[0120] The optimization goal of the WHERE encoding model only includes the subtasks WHERE-COL, WHERE-NUM, WHERE-OP, and WHERE-VALUE of the WHERE part, which are encoded using BERT to the corresponding text representation E W . According to E W Predict the WHERE subtask. First, predict the number of WHERE-NUM used by the column name COL. is the weight of the WHERE-NUM prediction module, and the formula is as follows:

[0121]

[0122] in, The classification probability distribution of the WHERE-NUM prediction module;

[0123] After predicting the number of column names used, you need to predict which column names are used for the corresponding number, that is, you need to select the column name COL to be used, and the value range is all column names in the current data table. is the weight of the WHERE-COL prediction module, and the classification probability distribution of the WHERE-COL prediction module The formula is as follows:

[0124]

[0125] After selecting the column name COL to be used, it is also necessary to predict the comparison operation WHERE-OP corresponding to the column name. The value range is {">", "<", "=", "!=", ">="", "<="}. Assume that the currently selected column name COL is the hth column in the data table T. i List, is the weight of the WHERE-OP prediction module, and the classification probability distribution of the WHERE-OP prediction module The formula is as follows:

[0126]

[0127] By performing sequence labeling on the original natural language query, the labeling contains <start>and <end>Two special tags, and the text between these two tags is the obtained conditional value; using this overall feature expression, a classification model is constructed for it to determine whether the candidate conditions composed of the VAL values of the current text type are in the SQL statement. The value range of the binary classification problem is {0, 1}. and are respectively WHERE-VAL start the prediction module and WHERE-VAL end the weight of the prediction module,

[0128]

[0129]

[0130] Among them, is WHERE-VAL start the probability distribution of the prediction module, is WHERE-VAL end the probability distribution of the prediction module.

[0131] 3.4 Construct the SELECT+WHERE part encoding model

[0132] The optimization objectives of the SELECT+WHERE encoding model include the subtasks SELECT-COL and SELECT-AGG of the SELECT part, and at the same time include the subtasks WHERE-COL, WHERE-NUM, WHERE-OP, and WHERE-VALUE of the WHERE part (SELECT COUNT(*)FROM SUBSTATION_BASIC WHERE OPERATE_STATE='returned' AND REGION='Qinghai Province' (the overall SQL statement) / count the number of substations that have been returned in Qinghai Province (natural language)). Use BERT to encode to obtain the corresponding text representation E SW . In addition, adopt the same prediction method as each subtask in steps 3.2 and 3.3 for classification prediction.

[0133] 3.5 Construct the multi-perspective decoupling layer for different tasks

[0134] After steps 3.2 and 3.3, the present invention obtains the text representations E S and E W for the SELECT part (the number of substations) and the WHERE part (substations returned in Qinghai Province); in addition, after step 3.4, the present invention obtains the text representations E SW for these two parts. On this basis, for E S and E W use E respectively SW Perform orthogonal projection and get E W1 and E S1 . E S1 To some extent, the WHERE part (Qinghai Province has been returned) feature is eliminated. Similarly, E W1 It also eliminates the impact of the SELECT part (number of substations) features on downstream tasks.

[0135] First, E S Projection to E SW Get E in the direction S* , the specific method is formulated as: E S* =Proj(E S ,E SW ),in, x and y are the corresponding eigenvectors.

[0136] Ask E again S With E S -E S* Projection:

[0137]

[0138] here It represents the purified vector E after eliminating the SELECT part (number of substations) features. W1 ; Similarly, we get E S1 .

[0139] Late Integration S and E S1 Get E S2 Perform the SELECT part (substation quantity) subtask prediction and integrate E W and E W1 Get E W2 Carry out prediction for the subtask in the WHERE part (the products have been returned to Qinghai Province).

[0140] E S2 =E S +E S1

[0141] E W2 =E W +E W1

[0142] In this way, the semantic feature vectors of the SELECT part (the number of substations) and the WHERE part (the substations have been withdrawn from service in Qinghai Province) are obtained from multiple perspectives.

[0143] 3.6 Constructing prediction layers for different subtasks

[0144] The feature vector E of the SELECT part (number of substations) and the WHERE part (Qinghai Province has withdrawn) obtained in step 3.5 S2 and E W2 For each subtask, the prediction of the subtask will follow the methods in steps 3.2 and 3.3.

[0145] Step 4: Train the NL2SQL model

[0146] The NL2SQL model constructed in step 3 is trained on the training set obtained in step 2, such as Figure 5 shown.

[0147] 4.1 Constructing the loss function

[0148] All of the above subtasks use classification models. During the model training process, the objective function of the model is the sum of the loss functions of all subtasks, and the parameters are updated by minimizing this sum of objective functions. This article uses the cross entropy loss function, which is the most commonly used in classification models, and the formula is as follows:

[0149]

[0150] Where N represents the number of categories, y j represents the label value of the jth class, which can be 0 or 1. j ) represents the probability value of the jth class prediction. The above formula expresses the loss function of a classification model. The overall objective function L of the model in this paper is the sum of all subtask loss functions.

[0151] 4.2 Building Optimization Functions

[0152] The deep learning models used in this article are all based on the pre-trained model BERT plus the task-specific classification layer fine-tuning training. In order to avoid destroying the representation ability of the original pre-trained model during the fine-tuning process, there is no need to train too many cycles, and the number of iterations is set to 10. Each deep learning model is implemented based on the deep learning framework Pytorch. During the training process, the learning rate size is set to 3e-5, the batch size is 128, the optimizer is AdamW, and the regularization method uses Dropout to prevent overfitting.

[0153] The model of the present invention achieves better results than the advanced models on the WikiSQL dataset, as shown in Table 1.

[0154] Table 1: Experimental results on WikiSQL.

[0155] Methods Logical from Accuracy Execution Accuracy Auxiliary Mapping Task 75.0 81.7 MQAN (unordered) 75.4 81.4 Execution-Guided Decoding 75.4 83.8 HydraNet(BERT_base) 78.95 85.30 Our model (BERT_base) 79.71 85.63

[0156] The model of the present invention is compared with some recent advanced methods, and the experimental results show that the method of the present invention is greatly improved.

[0157] The present application also provides a NL2SQL device based on multi-view feature decoupling, such as Figure 6 As shown: the device comprises:

[0158] The NL2SQL knowledge base construction unit uses the user's query questions to manually write SQL query statements, construct data entries that correspond to query questions, SQL query statements and table structures, and preliminarily build a knowledge base.

[0159] The NL2SQL data set construction unit is used to divide the training set, the validation set and the test set according to the NL2SQL knowledge base to construct the NL2SQL data set.

[0160] The NL2SQL model building unit is used to construct the semantic encoding models of the SELECT part (number of substations), the WHERE part (the substations that have been decommissioned in Qinghai Province) of the SQL query statement respectively; in order to effectively capture the semantic features of each part of SELECT and WHERE, a multi-perspective feature decoupling method is constructed.

[0161] NL2SQL model training unit is used to build loss function and optimization function.

[0162] Furthermore, the NL2SQL model building unit also includes:

[0163] The SELECT part encodes the model building unit. The optimization target of this unit only includes the subtasks SELECT-COL and SELECT-AGG of the SELECT part, which are encoded using BERT to the corresponding text representation E S , according to E S Predict the column name SELECT-COL and the aggregate operation SELECT-AGG.

[0164] The WHERE part encodes the model building unit. The optimization target of this unit only includes the subtasks WHERE-COL, WHERE-NUM, WHERE-OP and WHERE-VALUE of the WHERE part, which are encoded using BERT to the corresponding text representation E W , according to E W Predict the corresponding subtasks of the WHERE part.

[0165] The SELECT+WHERE part encoding model building unit, the optimization target of which includes the subtasks SELECT-COL and SELECT-AGG of the SELECT part, and the subtasks WHERE-COL, WHERE-NUM, WHERE-OP and WHERE-VALUE of the WHERE part, which are encoded using BERT to the corresponding text representation E SW In addition, the prediction is performed in the same way as the prediction of each subtask in the WHERE part encoding model building unit and the SELECT part encoding model building unit.

[0166] The multi-view decoupling construction unit for different tasks obtains the text feature vector E for the SELECT part and the WHERE part by the SELECT part encoding model construction unit and the WHERE part encoding model construction unit. S and E W In addition, after the SELECT+WHERE part coding model construction unit, the present invention obtains the text feature vector E for these two parts. SW On this basis, for E S and E W Use E SW Perform orthogonal projection and get E W1 and E S1 . E S1 To some extent, some features of WHERE are eliminated. Similarly, E W1 It also eliminates the impact of some SELECT features on downstream tasks.

[0167] E obtained by later integration S and E S1 Perform predictions on some of the SELECT subtasks and integrate the obtained E W and E E1 Perform prediction of the WHERE part of the subtask.

[0168] Different subtask prediction construction units use the feature vectors of the SELECT part and the WHERE part obtained in the different task multi-view decoupling construction units for the prediction of each subtask. The prediction of the subtask will follow the methods in the SELECT part encoding model construction unit and the WHERE part encoding model construction unit.

[0169] Furthermore, the NL2SQL model training unit further includes:

[0170] Loss function building unit, the objective function of the model is the sum of all subtask loss functions, and the objective function of this sum is minimized through model training;

[0171] The optimization function building unit is responsible for optimizing model parameters and adjusting the update of parameters in the model.

[0172] The above is only the technical solution of the present invention, and the protection scope of the present invention cannot be limited by this. For those skilled in the art, without departing from the principle of the present invention, the present invention can still be modified and replaced by equivalents, which all fall within the protection scope of the present invention.< / end> < / start> < / end> < / start>

Claims

1. A NL2SQL method based on multi-view feature decoupling, characterized in that: The method comprises the following steps: S1. Build the NL2SQL knowledge base, use the user's query questions, manually write SQL query statements, construct data entries that correspond to query questions, SQL query statements, and table structures, and form the NL2SQL knowledge base; S2. Construct the NL2SQL dataset. For each question, there is a matching SQL query statement and database table structure in the NL2SQL knowledge base. The training set, validation set, and test set are divided according to the NL2SQL knowledge base. S3. Build the NL2SQL model, sequentially build the input layer, build the SELECT part encoding model, build the WHERE part encoding model, build the SELECT+WHERE part encoding model, build the multi-view decoupling layer for different tasks, and build the prediction layer for different subtasks; The construction of the input layer in step S3 is specifically as follows: The input layer obtains user questions and table structures from the NL2SQL dataset and formalizes them as: Input:; The input layer uses BERT input, which consists of three parts: Token Embedding, Segment Embedding, and Position Embedding. The three vectors are added together to get the input vector of each word in the sentence. The input vector of each word in the sentence is combined into a matrix, which is the input of the sentence. The BERT input undergoes several multi-head attention mechanisms, layer normalization, and feedforward network layers to get the output: E=BERT(Input) The calculation formula for defining the classification probability distribution Pc for matrix E is: P c (E)=Softmax(Wtanh(E)) W represents a trainable parameter. Tanh is a common activation function in machine learning. The softmax function is used to normalize all values ​​in the vector and convert them into probability values. The sum of the normalized values ​​is 1. The construction of the SELECT part encoding model in step S3 is specifically as follows: The optimization goal of the SELECT encoding model only includes the subtasks SELECT-COL and SELECT-AGG (SELECTCOUNT(*) (SELECT part in SQL) of the SELECT part, and uses BERT to encode the corresponding text representation E S ; According to E S The predicted column name is SELECT-COL, and the value range is all column names in the current data table. is the weight of the SELECT-COL prediction module, and the classification probability distribution of the SELECT-COL prediction module The formula is as follows: After selecting the column name COL to be used, you also need to predict the aggregation operation SELECT-AGG corresponding to the column name. The value range is {"NONE", "MAX", "MIN", "COUNT", "SUM", "AVG"}. is the weight of the SELECT-AGG prediction module: in, The probability distribution of the SELECT-AGG prediction module; The construction of the WHERE part coding model in step S3 is specifically as follows: The optimization goal of the WHERE encoding model only includes the subtasks WHERE-COL, WHERE-NUM, WHERE-OP, and WHERE-VALUE of the WHERE part, which are encoded using BERT to the corresponding text representation E W , according to E W Predict the WHERE subtask. First, predict the number of WHERE-NUM used by the column name COL. is the weight of the WHERE-NUM prediction module, and the formula is as follows: in, The classification probability distribution of the WHERE-NUM prediction module; After predicting the number of column names used, you need to predict which column names are used for the corresponding number, that is, you need to select the column name COL to be used, and the value range is all column names in the current data table. is the weight of the WHERE-COL prediction module, and the classification probability distribution of the WHERE-COL prediction module The formula is as follows: After selecting the column name COL to be used, it is also necessary to predict the comparison operation WHERE-OP corresponding to the column name. The value range is {">", "<", "=", "!=", ">="", "<="}. Assume that the currently selected column name COL is the hth column in the data table T. i List, is the weight of the WHERE-OP prediction module, and the classification probability distribution of the WHERE-OP prediction module The formula is as follows: By performing sequence labeling on the original natural language query, the labeling contains <start>and <end>Two special tags, the text between the two tags is the obtained conditional value; using this overall feature expression, a classification model is constructed to determine whether the candidate condition constituted by the VAL value of the current text type is in the SQL statement. The value range of the binary classification problem is {0, 1}. as well as are WHERE-VAL start Prediction Module and WHERE-VAL end The weight of the prediction module,< / end> < / start> in, For WHERE-VAL start The probability distribution of the prediction module, For WHERE-VAL end Probability distribution of prediction modules; The coding model for constructing the SELECT+WHERE part is specifically as follows: The optimization target of the SELECT+WHERE encoding model includes the subtasks SELECT-COL and SELECT-AGG of the SELECT part, and the subtasks WHERE-COL, WHERE-NUM, WHERE-OP, and WHERE-VALUE of the WHERE part, which are encoded using BERT to the corresponding text representation E SW , and classify and predict each subtask; Constructing multi-view decoupling layers for different tasks is as follows: Based on the text representation E obtained for the SELECT part S , the text representation of the WHERE part W , and the text representation of the SELECT part and the WHERE part SW On this basis, for E S and E W Use E SW Perform orthogonal projection and get E W1 and E S1 : First, E S Projection to E SW Get E in the direction S* , the specific method is formulated as: E S* =Proj(E S , E SW ),in, x and v are the corresponding eigenvectors; Ask E again S With E S -E S* Projection here It represents the purified vector E after eliminating some SELECT features. W1 ; Similarly, we get E S1 ; Late Integration S and E S1 Get E S2 Perform predictions on some of the SELECT subtasks and integrate E W and E W1 Get E W2 Perform prediction of some subtasks in WHERE, that is AND S2 =And S +E S1 AND W2 =And W +E W1 In this way, the semantic feature vectors of the SELECT part and the WHERE part are obtained from multiple perspectives; Constructing different subtask prediction layers is as follows: The semantic feature vector E of the SELECT part and the WHERE part is obtained S2 and E W2 Used for prediction of each subtask; S4. NL2SQL model training: Use the training set obtained in step S2 to train the NL2SQL model constructed in step S3, so as to construct a loss function and an optimization function.

2. The NL2SQL method based on multi-view feature decoupling according to claim 1, characterized in that: Training the NL2SQL model is as follows: Construct the loss function: Using the cross entropy loss function, the formula for the cross entropy loss function loss(P(x), y) of a classification model is as follows: Where N represents the number of categories, y j represents the label value of the jth class, which can be 0 or 1. j ) represents the probability value of the j-th category prediction, and the overall objective function L of the model is the sum of all subtask loss functions; Constructing the optimization function: The deep learning model used is based on the pre-trained model BERT plus task-specific classification layer fine-tuning. The specific classification layer fine-tuning is performed during the fine-tuning process, and the number of iterations is set to 10. Each deep learning model is implemented based on the deep learning framework Pytorch. During the training process, the learning rate size set is 3e-5, the batch size is 128, the optimizer is AdamW, and the regularization method uses Dropout to prevent overfitting.

3. A NL2SQL implementation device based on multi-view feature decoupling using any one of the methods of claim 1 or 2, characterized in that: The device comprises: The NL2SQL knowledge base construction unit is used to use the user's query questions, manually write SQL query statements, construct data entries that correspond to query questions, SQL query statements and table structures, and preliminarily build a knowledge base; The NL2SQL data set construction unit is used to divide the training set, the validation set and the test set according to the NL2SQL knowledge base to construct the NL2SQL data set; NL2SQL model building unit, used to build the semantic coding model of the SELECT part, the WHERE part and the SQL query statement as a whole; NL2SQL model training unit, used to build loss function and optimization function, including: Loss function building unit, the objective function of the model is the sum of all subtask loss functions, and the model is trained to minimize this sum of objective functions; The optimization function building unit is responsible for optimizing model parameters and adjusting the update of parameters in the model.

4. The NL2SQL implementation device based on multi-view feature decoupling according to claim 3 is characterized in that: The NL2SQL model building unit also includes: The SELECT part encodes the model building unit. The optimization target of this unit only includes the subtasks SELECT-COL and SELECT-AGG of the SELECT part, which are encoded using BERT to the corresponding text representation E S , according to E S Predict the column name SELECT-COL and the aggregation operation SELECT-AGG; The WHERE part encodes the model building unit. The optimization target of this unit only includes the subtasks WHERE-COL, WHERE-NUM, WHERE-OP and WHERE-VALUE of the WHERE part, which are encoded using BERT to the corresponding text representation E W , according to E W Predict the corresponding subtasks of the WHERE part; The SELECT+WHERE part encoding model building unit, the optimization target of which includes the subtasks SELECT-COL and SELECT-AGG of the SELECT part, and the subtasks WHERE-COL, WHERE-NUM, WHERE-OP and WHERE-VALUE of the WHERE part, which are encoded using BERT to the corresponding text representation E SW ; In addition, the same prediction method as each subtask in the WHERE part coding model construction unit and the SELECT part coding model construction unit is adopted to make predictions; The multi-view decoupling construction unit for different tasks obtains the text feature vector E for the SELECT part and the WHERE part by the SELECT part encoding model construction unit and the WHERE part encoding model construction unit. S and E W ; In addition, after the SELECT+WHERE part encoding model construction unit, the text feature vector E for these two parts is obtained SW On this basis, for E S and E W Use E SW Perform orthogonal projection and get E W1 and E S1 ; E obtained by later integration S and E S1 Perform predictions on some of the SELECT subtasks and integrate the obtained E W and E W1 Perform prediction on some subtasks in WHERE; Different subtask prediction construction units are used to use the feature vectors of the SELECT part and the WHERE part obtained in the different task multi-view decoupling construction units for the prediction of each subtask. The prediction of the subtask will follow the methods in the SELECT part encoding model construction unit and the WHERE part encoding model construction unit.

Images