A Multi-Task NL2SQL Method Based on a Two-Level Multi-Gated Expert Hybrid Model

Abstract

This invention discloses a multi-task NL2SQL method based on a two-layer multi-gated expert hybrid model (MMOE), comprising the following steps: Step 1, an encoding layer construction step, used to obtain the embedded representations of the question and table schema; Step 2, a multi-gated recurrent neural network hybrid layer construction step, used to further extract different types of semantic information using the multi-gated expert hybrid model; Step 3, a multi-gated attention pooling hybrid layer construction step, used to obtain the vector representation of the question under different sub-tasks using the multi-gated expert hybrid model; Step 4, a sub-task learning layer construction step, used to predict each sub-task; and Step 5, a multi-task NL2SQL model training step, used to construct the loss function and the optimization function. This invention shows a particularly significant improvement in query matching accuracy, indicating that the model of this invention has a significant performance improvement on NL2SQL datasets.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, natural language processing, and multi-task learning, specifically to a multi-task NL2SQL method based on a two-layer multi-gated expert hybrid model. Technical Background

[0002] Artificial intelligence is an important branch of computer science, mainly including image recognition, natural language processing, knowledge graphs and other research areas. In recent years, with the continuous improvement of computing power and the increasing amount of data acquired, the field of artificial intelligence has made rapid progress and has been widely applied, greatly promoting the deep integration of emerging industries and has initially met the conditions for practical application in various fields.

[0003] Relational databases are ubiquitous in many practical applications. However, they typically require the use of structured query languages ​​(such as SQL) to interact with them. Text-to-SQL tasks involve translating natural language descriptions into corresponding SQL queries, allowing users to interact directly with relational databases using natural language. This technology effectively assists in querying massive databases and has broad application prospects, such as database-based automated question answering. Summary of the Invention

[0004] Purpose of the Invention: NL2SQL mainly involves two aspects of prediction: SELECT and WHERE. Learning the correlation and differences between SELECT and WHERE is a core problem in NL2SQL. This application addresses this problem using deep learning technology. To learn the correlation and differences between subtasks, a multi-task learning model based on a two-layer multi-gated expert hybrid model is proposed. The proposed model achieves significant results on real NL2SQL datasets and has great application potential in the field of automatic question answering systems.

[0005] Technical solution: To achieve the above objectives, the technical solution adopted by this invention is as follows:

[0006] A multi-task NL2SQL method based on a two-layer multi-gated expert hybrid model includes the following steps:

[0007] Step 1: Encoding layer construction step, used to obtain the embedded representation of the problem and table schema.

[0008] Step 2: Construction of a multi-gated recurrent neural network hybrid layer, used to further extract different types of semantic information using a multi-gated expert hybrid model.

[0009] Step 3: Construction of a multi-gated attention pooling hybrid layer, used to obtain vector representations of the problem under different subtasks using a multi-gated expert hybrid model.

[0010] Step 4: Sub-task learning layer construction step, used to predict each sub-task.

[0011] Step 5: Multi-task NL2SQL model training steps, used to construct the loss function and the optimization function.

[0012] The model of this invention was compared with some recent state-of-the-art methods, and experimental results show that the method of this invention represents a significant improvement. The improvement in query matching accuracy is particularly significant, indicating that the model of this invention has a substantial performance improvement on NL2SQL datasets.

[0013] Terminology Explanation:

[0014] Multi-task learning is a subfield of machine learning that aims to leverage the similarities between different tasks to solve multiple different tasks simultaneously. This can improve learning efficiency and accuracy.

[0015] The two-layer multi-gated expert hybrid model is a classic multi-task learning model. It sets up multiple expert systems at the bottom layer, each with its own area of ​​expertise. At the same time, it sets up a gating system to generate a set of weights for each subtask. In this way, each subtask can use the weighted sum of the outputs of different expert systems as its own input according to its own characteristics. This model can effectively learn the correlation and differences between different subtasks.

[0016] RoBERTa: RoBERTa is an improved version of BERT, a pre-trained language representation model. It builds upon BERT's language masking strategy, modifying key hyperparameters in BERT, including removing the next-sentence pre-training objective and using a larger batch size and learning rate. RoBERTa also undergoes an order of magnitude more training than BERT, taking significantly longer. This allows RoBERTa to generalize to downstream tasks better than BERT.

[0017] Each implementation step of the method is as follows:

[0018] The specific steps are as follows:

[0019] The specific steps for building a multi-task NL2SQL model are as follows:

[0020] Step 1: The specific steps for constructing the coding layer are as follows:

[0021] This invention uses RoBERTa as the encoder, inputting each column of the table and the question into the encoder separately. After several multi-head attention, layer normalization, and feedforward network layers, the output is obtained as a representation of the question and table pattern.

[0022]

[0023] in, For the sake of columns and natural language problems, for Pre-trained language models This represents the question and table schema.

[0024] Step 2: The specific steps for constructing a multi-gated recurrent neural network hybrid layer are as follows:

[0025] First, this invention designs an expert system that uses a bidirectional LSTM (BiLSTM) recurrent neural network to further obtain semantic information about the question and table patterns:

[0026]

[0027] in Let i be the vector representation of the i-th token. It is a bidirectional LSTM (BiLSTM) recurrent neural network. This is the vector representation of the i-th token after passing through a bidirectional LSTM (BiLSTM) recurrent neural network.

[0028] To acquire different types of semantic information, this invention sets up three expert systems, each capable of learning within its own domain of expertise. The output of the encoding layer is simultaneously input into all three expert systems to acquire different types of semantic information.

[0029]

[0030]

[0031]

[0032] in Let i be the vector representation of the i-th token. It is a bidirectional LSTM (BiLSTM) recurrent neural network. The representation of the problem and table pattern is the representation after passing through the i-th bidirectional LSTM (BiLSTM) recurrent neural network.

[0033] Subsequently, this invention designs a gating system to integrate the outputs of three expert systems. This involves generating three sets of weights, each with three values, and then weighted summing the outputs of the three expert systems to obtain three new outputs. This approach allows the three independent expert systems to learn different types of semantic information while also learning the correlations between them. Since RoBERTa's pooled output vector p can effectively integrate the information of the input question and columns, this invention utilizes this pooled output vector, multiplying it by three trainable weight matrices, and then passing it through a Softmax layer to obtain three weight vectors. Each value in each weight vector represents the weight of the result from each expert system. These weighted sums yield three new semantic representation matrices of the question and table patterns, which are then used as input to the next layer.

[0034]

[0035]

[0036] in, This is the pooled output vector of RoBERTa. Let be the i-th trainable weight matrix. for function, For the i-th weight vector, For the j-th value of the i-th weight vector, This is the input matrix for the i-th expert system in the next layer (i.e., the multi-gated attention pooling hybrid layer).

[0037] Step 3: The specific steps for constructing the multi-gated attention pooling hybrid layer are as follows:

[0038] First, this invention designs a pooled expert system that uses an attention mechanism to perform a weighted summation of the previous layer's representation to obtain the vector representation of the problem. It first initializes a trainable vector as the query vector Q for the attention mechanism, then uses the output of the previous layer passed through a single-layer neural network as the key vector K, ultimately obtaining the final vector representation of the problem.

[0039] )

[0040]

[0041]

[0042] in, As weight, For deviation, Let be the representation vector of the j-th token in the input of the i-th expert system. for function, Let be the key vector of the j-th token in the i-th expert system. For trainable query vectors, for function, Let J be the weight of the j-th token in the i-th expert system. Let be the output vector of the i-th expert system.

[0043] Subsequently, this invention uses three such pooling expert systems, inputting the three outputs of the first layer into their respective pooling expert systems to obtain three different types of vector representations. Furthermore, similar to the gating system described above, this invention utilizes the pooling output vector of RoBERTa, multiplies it by seven trainable weight matrices, and then passes it through a Softmax layer to obtain seven weight vectors. Each value in each weight vector represents the weight of the result of each expert system under each subtask. These weighted sums are then used to obtain the inputs for the seven subtasks.

[0044]

[0045] :

[0046] in, This is the pooled output vector of RoBERTa. Let be the i-th trainable weight matrix. for function, For the i-th weight vector, For the j-th value of the i-th weight vector, Let be the input vector for the i-th subtask.

[0047] Step 4: The specific steps for constructing the sub-task learning layer are as follows:

[0048] This invention comprises 7 sub-tasks, namely:

[0049] The SELECT section has two subtasks: predicting the column names of the SELECT (SELECT-COL) and predicting the operations of the SELECT (SELECT-AGG).

[0050] The WHERE part has four sub-tasks: predicting the column name of WHERE (WHERE-COL), predicting the number of WHERE columns (WHERE-NUM), predicting the operation of WHERE (WHERE-OP), and predicting the value of WHERE (WHERE-VALUE).

[0051] And predict the column name RELEVANCE-COL that will appear in the SQL statement, that is, predict the column name that will appear in SELECT-COL or WHERE-COL, which is used to combine the tasks of SELECT and WHERE.

[0052] First, predict the number of times the column name COL is used (WHERE-NUM), using the following formula:

[0053]

[0054] in As weight, For deviation, This represents a quantity probability distribution.

[0055] After predicting the number of column names COL, the next step is to predict which column names are used for the corresponding number in WHERE-COL, with the value range being all column names in the current data table. The formula is as follows:

[0056]

[0057] in As weight, For deviation, This represents the probability distribution for classification.

[0058] After selecting the column name COL, it is also necessary to predict the corresponding comparison operation WHERE-OP, with values ​​ranging from {">", "<", "=", "!=", ">=", "<="}. Assume the currently selected column name COL is from table T. Classification probability distribution The formula is as follows:

[0059]

[0060] in As weight, For deviation, Let be the probability that the i-th column in the data table is selected.

[0061] By performing sequence labeling on the original natural language query, the labeling contains... <start>and <end>Two special labels are used, and the text between these two labels represents the retrieved condition values. Using this overall feature representation, a classification model is built to determine whether the candidate conditions formed by the VAL values ​​of the current text type are present in the SQL statement.

[0062] )

[0063] )

[0064]

[0065]

[0066] in, , As weight, , For deviation, This is a representation of the START message. This is a representation of the END information. for function, for <start>Predicted location of the tag for <end>Predicted location of the label.

[0067] Next, predict the column name using SELECT-COL, where the value range is all column names in the current data table, and calculate the classification probability distribution. The formula is as follows:

[0068]

[0069] in As weight, For deviation, This represents the probability distribution for classification.

[0070] After selecting the column name COL, you also need to perform a predictive aggregation operation SELECT-AGG, with the value range being {"NONE", "MAX", "MIN", "COUNT", "SUM", "AVG"}.

[0071]

[0072] in As weight, For deviation, This represents the classification probability distribution for aggregation operations.

[0073] Finally, predict RELEVANCE-COL, with a value ranging from all column names in the current data table, and follow the classification probability distribution. The formula is as follows:

[0074] .

[0075] in As weight, For deviation, For classification probability distribution

[0076] Step 5: The specific steps for training the multi-task NL2SQL model are as follows:

[0077] Construct the loss function:

[0078] All of the above sub-tasks use classification models. During model training, the objective function of the model is the sum of the loss functions of all sub-tasks. The parameters are updated by minimizing this sum of objective functions.

[0079] This invention employs the cross-entropy loss function, most commonly used in classification models, as shown in the following formula:

[0080] (1)

[0081] in Indicates the number of categories. Indicates the first The class's label value can be either 0 or 1. Indicates the first The probability value of the class prediction. Construct an optimization function for the sum of the loss functions:

[0082] The deep learning models used in this paper are all based on the pre-trained RoBERTa model, fine-tuned with a task-specific classification layer. To avoid damaging the representational capabilities of the original pre-trained model during fine-tuning, the number of training epochs is minimized, with 5 iterations per epoch. All deep learning models are implemented using the PyTorch framework. During training, the learning rate is set to 3e-5, the batch size to 256, the optimizer to AdamW, and Dropout is used for regularization to prevent overfitting.

[0083] The model of this invention was compared with some recent state-of-the-art methods, and experimental results show that the method of this invention represents a significant improvement. The improvement in query matching accuracy is particularly significant, indicating that the model of this invention has a substantial performance improvement on NL2SQL datasets.

[0084] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0085] Technical effects:

[0086] 1) The expert system proposed in this invention can effectively obtain the vector representation of the problem.

[0087] 2) This invention proposes to use a two-layer multi-gated expert hybrid model and a multi-task learning model to learn the correlation and differences between SELECT and WHERE.

[0088] 3) This invention defines and implements a complete multi-task NL2SQL method. Experimental results on the WikiSQL dataset show that, compared with existing methods, this invention improves the query matching accuracy by 0.2%. Attached Figure Description

[0089] Figure 1 A flowchart illustrating the construction steps of a multi-task NL2SQL method based on a two-layer multi-gated expert hybrid model;

[0090] Figure 2 This is a schematic diagram of the model structure of a multi-task NL2SQL method based on a two-layer multi-gated expert hybrid model.

[0091] Figure 3 This is a schematic diagram of a hybrid layer model of a multi-gated recurrent neural network.

[0092] Figure 4 This is a schematic diagram of the multi-gated attention pooling hybrid layer model structure.

[0093] Figure 5 This is a schematic diagram of the overall model structure of a hybrid layer of multi-gated recurrent neural networks and a hybrid layer of multi-gated attention pooling. Detailed Implementation

[0094] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these examples are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0095] Example 1: As Figure 1 As shown, a multi-task NL2SQL method based on a two-layer multi-gated expert hybrid model includes the following steps:

[0096] Step 1: Construct the encoding layer.

[0097] This invention uses RoBERTa as the encoder, inputting each column of the table and the question into the encoder separately. After several multi-head attention, layer normalization, and feedforward network layers, the output is obtained as a representation of the question and table pattern.

[0098]

[0099] in, For the sake of columns and natural language problems, for Pre-trained language models This represents the question and table schema.

[0100] Step 2: Construct a hybrid layer of multi-gated recurrent neural networks.

[0101] First, this invention designs an expert system that uses a bidirectional LSTM (BiLSTM) recurrent neural network to further obtain semantic information about the question and table patterns:

[0102]

[0103] in Let i be the vector representation of the i-th token. It is a bidirectional LSTM (BiLSTM) recurrent neural network. This is the vector representation of the i-th token after passing through a bidirectional LSTM (BiLSTM) recurrent neural network.

[0104] To acquire different types of semantic information, this invention sets up three expert systems, each capable of learning within its own domain of expertise. The output of the encoding layer is simultaneously input into all three expert systems to acquire different types of semantic information.

[0105]

[0106]

[0107]

[0108] in Let i be the vector representation of the i-th token. It is a bidirectional LSTM (BiLSTM) recurrent neural network. The representation of the problem and table pattern is the representation after passing through the i-th bidirectional LSTM (BiLSTM) recurrent neural network.

[0109] Subsequently, this invention designs a gating system to integrate the outputs of three expert systems. This involves generating three sets of weights, each with three values, and then weighted summing the outputs of the three expert systems to obtain three new outputs. This approach allows the three independent expert systems to learn different types of semantic information while also learning the correlations between them. Since RoBERTa's pooled output vector p can effectively integrate the information of the input question and columns, this invention utilizes this pooled output vector, multiplying it by three trainable weight matrices, and then passing it through a Softmax layer to obtain three weight vectors. Each value in each weight vector represents the weight of the result from each expert system. These weighted sums yield three new semantic representation matrices of the question and table patterns, which are then used as input to the next layer.

[0110]

[0111]

[0112] in, This is the pooled output vector of RoBERTa. Let be the i-th trainable weight matrix. for function, For the i-th weight vector, For the j-th value of the i-th weight vector, This is the input matrix for the i-th expert system in the next layer (i.e., the multi-gated attention pooling hybrid layer).

[0113] The structure of a hybrid layer in a multi-gated recurrent neural network is as follows: Figure 3 As shown:

[0114] Step 3: Construct a multi-gated attention pooling hybrid layer

[0115] First, this invention designs a pooled expert system that uses an attention mechanism to perform a weighted summation of the previous layer's representation to obtain the vector representation of the problem. It first initializes a trainable vector as the query vector Q for the attention mechanism, then uses the output of the previous layer passed through a single-layer neural network as the key vector K, ultimately obtaining the final vector representation of the problem.

[0116] )

[0117]

[0118]

[0119] in, As weight, For deviation, Let be the representation vector of the j-th token in the input of the i-th expert system. for function, Let be the key vector of the j-th token in the i-th expert system. For trainable query vectors, for function, Let J be the weight of the j-th token in the i-th expert system. Let be the output vector of the i-th expert system.

[0120] Subsequently, this invention uses three such pooling expert systems, inputting the three outputs of the first layer into their respective pooling expert systems to obtain three different types of vector representations. Furthermore, similar to the gating system described above, this invention utilizes the pooling output vector of RoBERTa, multiplies it by seven trainable weight matrices, and then passes it through a Softmax layer to obtain seven weight vectors. Each value in each weight vector represents the weight of the result of each expert system under each subtask. These weighted sums are then used to obtain the inputs for the seven subtasks.

[0121]

[0122]

[0123] in, This is the pooled output vector of RoBERTa. Let be the i-th trainable weight matrix. for function, For the i-th weight vector, For the j-th value of the i-th weight vector, Let be the input vector for the i-th subtask.

[0124] The structure of the multi-gated attention pooling hybrid layer is as follows: Figure 4 As shown:

[0125] The overall structure of the hybrid layer of multi-gated recurrent neural networks and the hybrid layer of multi-gated attention pooling is as follows: Figure 5 As shown,

[0126] Step 4: Construct the sub-task learning layer.

[0127] This invention comprises 7 sub-tasks, namely:

[0128] The SELECT section has two subtasks: predicting the column names of the SELECT (SELECT-COL) and predicting the operations of the SELECT (SELECT-AGG).

[0129] The WHERE part has four sub-tasks: predicting the column name of WHERE (WHERE-COL), predicting the number of WHERE columns (WHERE-NUM), predicting the operation of WHERE (WHERE-OP), and predicting the value of WHERE (WHERE-VALUE).

[0130] And predict the column name RELEVANCE-COL that will appear in the SQL statement, that is, predict the column name that will appear in SELECT-COL or WHERE-COL, which is used to combine the tasks of SELECT and WHERE.

[0131] First, predict the number of times the column name COL is used (WHERE-NUM), using the following formula:

[0132]

[0133] in As weight, For deviation, This represents a quantity probability distribution.

[0134] After predicting the number of column names COL, the next step is to predict which column names are used for the corresponding number in WHERE-COL, with the value range being all column names in the current data table. The formula is as follows:

[0135]

[0136] in As weight, For deviation, This represents the probability distribution for classification.

[0137] After selecting the column name COL, it is also necessary to predict the corresponding comparison operation WHERE-OP, with values ​​ranging from {">", "<", "=", "!=", ">=", "<="}. Let's assume the currently selected column name COL is from table T. Classification probability distribution The formula is as follows:

[0138]

[0139] in As weight, For deviation, Let be the probability that the i-th column in the data table is selected.

[0140] By performing sequence labeling on the original natural language query, the labeling contains... <start>and <end>Two special labels are used, and the text between these two labels represents the retrieved condition values. Using this overall feature representation, a classification model is built to determine whether the candidate conditions formed by the VAL values ​​of the current text type are present in the SQL statement.

[0141] )

[0142] )

[0143]

[0144]

[0145] in, , As weight, , For deviation, This is a representation of the START message. This is a representation of the END information. for function, for <start>Predicted location of the tag for <end>Predicted location of the label.

[0146] Next, predict the column name using SELECT-COL, where the value range is all column names in the current data table, and calculate the classification probability distribution. The formula is as follows:

[0147]

[0148] in As weight, For deviation, This represents the probability distribution for classification.

[0149] After selecting the column name COL, you also need to perform a predictive aggregation operation SELECT-AGG, with the value range being {"NONE", "MAX", "MIN", "COUNT", "SUM", "AVG"}.

[0150]

[0151] in As weight, For deviation, This represents the classification probability distribution for aggregation operations.

[0152] Finally, predict RELEVANCE-COL, with a value ranging from all column names in the current data table, and follow the classification probability distribution. The formula is as follows:

[0153] .

[0154] in As weight, For deviation, For classification probability distribution

[0155] Step 5: Train the multi-task NL2SQL model.

[0156] Constructing the loss function

[0157] All of the above sub-tasks use classification models. During model training, the objective function of the model is the sum of the loss functions of all sub-tasks. The parameters are updated by minimizing this sum of objective functions.

[0158] This invention employs the cross-entropy loss function, most commonly used in classification models, as shown in the following formula:

[0159] (1)

[0160] in Indicates the number of categories. Indicates the first The class's label value can be either 0 or 1. Indicates the first The probability value of the class prediction. The sum of the loss functions

[0161] Construct optimization function

[0162] The deep learning models used in this paper are all based on the pre-trained RoBERTa model, fine-tuned with a task-specific classification layer. To avoid damaging the representational capabilities of the original pre-trained model during fine-tuning, the number of training epochs is minimized, with 5 iterations per epoch. All deep learning models are implemented using the PyTorch framework. During training, the learning rate is set to 3e-5, the batch size to 256, the optimizer to AdamW, and Dropout is used for regularization to prevent overfitting.

[0163] The model of this invention achieved better results than state-of-the-art models on the WikiSQL dataset, as shown in Table 1.

[0164] Table 1: Experimental results on WikiSQL.

[0165] Methods Logical from Accuracy Execution Accuracy HydraNet (RoBERTa_base+EG) 86.5 92.2 Our model (RoBERTa_base+EG) 86.7 92.4

[0166] The model of this invention was compared with some recent state-of-the-art methods, and experimental results show that the method of this invention represents a significant improvement. The improvement in query matching accuracy is particularly significant, indicating that the model of this invention has a substantial performance improvement on NL2SQL datasets.

[0167] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.< / end> < / start> < / end> < / start> < / end> < / start> < / end> < / start>

Claims

1. A multi-task NL2SQL method based on a two-layer multi-gated expert hybrid model, characterized in that, The method includes the following steps: Step 1: Encoding layer construction step, used to obtain the embedded representation of the question and table schema; Step 2: Construction of multi-gated recurrent neural network hybrid layer, used to further extract different types of semantic information using a multi-gated expert hybrid model; Step 3: Construction of a multi-gated attention pooling hybrid layer, used to leverage a multi-gated expert hybrid model. Obtain vector representations of the problems under different subtasks; Step 4: Subtask learning layer construction step, used to predict each subtask; Step 5: Multi-task NL2SQL model training steps, used to construct the loss function and the optimization function; Specifically, the construction steps of the multi-gated recurrent neural network hybrid layer in step 2 are as follows: First, we design an expert system that uses a bidirectional LSTM (BiLSTM) recurrent neural network to further extract semantic information about the question and table patterns: (z1,...,z n+m )=BiLSTM(h1,...,h n+m ) Where h i Let z be the vector representation of the i-th token. BiLSTM is a bidirectional LSTM recurrent neural network. i This is the vector representation of the i-th token after passing through a bidirectional LSTM (BiLSTM) recurrent neural network; To acquire different types of semantic information, three expert systems are set up, each learning its own area of ​​expertise. The output of the encoding layer is simultaneously input into the three expert systems to acquire different types of semantic information. Z1=BiLSTM(h1,...,h n+m ) Z2=BiLSTM(h1,...,h n+m ) Z3=BiLSTM(h1,...,h n+m ) Where h i Let Z be the vector representation of the i-th token. BiLSTM is a bidirectional LSTM recurrent neural network. i The representation of the problem and table pattern after passing through the i-th bidirectional LSTM recurrent neural network; Next, a gating system is designed to integrate the outputs of the three expert systems. This involves generating three sets of weights, each with three values. The weighted sum of the outputs from the three expert systems yields three new outputs. This allows the three independent expert systems to learn different types of semantic information while also learning the correlations between them. Since RoBERTa's pooling output vector p can integrate information from the input question and columns, this pooling output vector is multiplied by three trainable weight matrices. After passing through a Softmax layer, three weight vectors are obtained. Each value in each weight vector represents the weight of the result from each expert system. These weighted sums yield three new semantic representation matrices of the question and table patterns, which are then used as input to the next layer. g1 i =Softmax(pW g1i ) E1 i =∑g1 ij WITH j Where p is the pooling output vector of RoBERTa, and W g1i Let g1 be the i-th trainable weight matrix, Softmax be the Softmax function, and g1 be the weight matrix. i For the i-th weight vector, g1 ij E1 is the j-th value of the i-th weight vector. i This is the input matrix for the i-th expert system in the next layer, namely the multi-gated attention pooling hybrid layer.

2. The multi-task NL2SQL method based on a two-layer multi-gated expert hybrid model according to claim 1, characterized in that, The specific steps for constructing the coding layer in step 1 are as follows: Using RoBERTa as the encoder, each column of the table is input separately along with the question into the encoder. After several passes of multi-head attention, layer normalization, and feedforward network layers, the output is obtained, which serves as a representation of the question and table pattern. H = RoBERTa(Input); Where Input represents the columns and the natural language question, RoBERTa represents the RoBERTa pre-trained language model, and H represents the question and table pattern.

3. The multi-task NL2SQL method based on a two-layer multi-gated expert hybrid model according to claim 2, characterized in that: The specific steps for constructing the multi-gated attention pooling hybrid layer in step 3 are as follows: First, a pooling expert system is designed, using an attention mechanism to perform a weighted summation of the previous layer's representation to obtain the problem's vector representation. A trainable vector is initialized as the query vector Q for the attention mechanism. The output of the previous layer is then processed through a single-layer neural network as the key vector K, ultimately yielding the final problem's vector representation h. c k ij =ReLU(W k E1 ij +b k ) a in+1 ,...,a in+m =softmax(q * k in+1 ,...,q * k in+m ) E2 i =∑a ij E1 ij Among them, W k As the weight, b k For deviation, E1 ij Let k be the representation vector of the j-th token in the input of the i-th expert system, and ReLU be the ReLU function. ij Let be the key vector of the j-th token in the i-th expert system, q be a trainable query vector, softmax be the softmax function, and a be the key vector of the j-th token in the i-th expert system. ij Let E2 be the weight of the j-th token in the i-th expert system. i Let i be the output vector of the i-th expert system; Then, using three such pooling expert systems, the three outputs of the first layer are respectively input into the corresponding pooling expert systems to obtain three different types of vector representations. Using the pooling output vector of RoBERTa, it is multiplied with seven trainable weight matrices, and after passing through the Softmax layer, seven weight vectors are obtained. Each value of each weight vector is the weight of the result of each expert system under each subtask. The weighted sum is then used to obtain the input of the seven subtasks. g2 i =Softmax(pW g2i ) task_input i =∑g2 ij E2 j ; Where p is the pooling output vector of RoBERTa, and W g2i Let g be the i-th trainable weight matrix, Softmax be the Softmax function, and g2 be the weight matrix. i For the i-th weight vector, g2 ij For the j-th value of the i-th weight vector, task_input i Let be the input vector for the i-th subtask.

4. The multi-task NL2SQL method based on a two-layer multi-gated expert hybrid model according to claim 3, characterized in that: The specific steps for constructing the sub-task learning layer in step 4 are as follows: It contains 7 sub-tasks, namely: The SELECT section has two subtasks: predicting the column names in the SELECT (SELECT-COL) and predicting the operations in the SELECT (SELECT-AGG). The WHERE part has four sub-tasks: predicting the column name of WHERE (WHERE-COL), predicting the number of WHERE columns (WHERE-NUM), predicting the operation of WHERE (WHERE-OP), and predicting the value of WHERE (WHERE-VALUE). And it predicts the column names that will appear in the SQL statement using RELEVANCE-COL, that is, predicts the column names that will appear in SELECT-COL or WHERE-COL, which is used to combine the tasks of SELECT and WHERE. First, predict the number of times the column name COL is used (WHERE-NUM), using the following formula: Among them W num As the weight, b num For deviation, It is a quantity probability distribution; After predicting the number of column names COL, the next step is to predict which column names are used for the corresponding number in WHERE-COL, with the value range being all column names in the current data table. The formula is as follows: Among them W wcol As the weight, b wcol For deviation, For classification probability distribution; After selecting the column name COL, it is also necessary to predict the corresponding comparison operation WHERE-OP, with values ​​ranging from {">", "<", "=", "!=", ">=", "<=". Assuming the currently selected column name COL is h in data table T... i Classification probability distribution The formula is as follows: Among them W op As the weight, b op For deviation, Let be the probability that the i-th column in the data table is selected; By performing sequence labeling on the original natural language query, the labeling contains... <start>and <end> Two special labels are used, and the text between these two labels represents the retrieved condition values. Using this overall feature representation, a classification model is built to determine whether the candidate conditions formed by the VAL values ​​of the current text type are present in the SQL statement.< / end> < / start> h start =ReLU(task_input i W s +b s ) h end =ReLU(task_input i W e +b e ) p start =argmax(h start *h n+1 ,...,h start *h n+m ) p end =argmax(h end *h n+1 ,...,h end *h n+m ) Among them, W s W e As the weight, b s b e For deviation, h start For the representation of START information, h end This represents the END information, argmax is the argmax function, and p start for <start>The predicted location of the label, p end for <end> Predicted location of the tag;< / end> < / start> Next, predict the column name using SELECT-COL, where the value range is all column names in the current data table, and calculate the classification probability distribution. The formula is as follows: Among them W scol As the weight, b scol For deviation, For classification probability distribution; After selecting the column name COL, you also need to perform a predictive aggregation operation SELECT-AGG, with the value range being {"NONE", "MAX", "MIN", "COUNT", "SUM", "AVG"}. Among them W agg As the weight, b agg For deviation, The classification probability distribution for aggregation operations; Finally, predict RELEVANCE-COL, with a value ranging from all column names in the current data table, and follow the classification probability distribution. The formula is as follows: Among them W rcol As the weight, b rcol For deviation, This represents the probability distribution for classification.

5. The multi-task NL2SQL method based on a two-layer multi-gated expert hybrid model according to claim 4, characterized in that: The specific steps for training the multi-task NL2SQL model in step 5 are as follows: Construct the loss function, All the subtasks described above use classification models. During model training, the model's objective function is the sum of the loss functions of all subtasks. Parameters are updated by minimizing this sum of objective functions. The cross-entropy loss function, most commonly used in classification models, is adopted, and the formula is as follows: Where N represents the number of categories, y j This represents the label value of the j-th class, which can be either 0 or 1, P(x j Let y represent the probability value predicted for the j-th class, and loss(p(x), y) be the sum of the loss functions. Construct an optimization function. The deep learning models used were all based on the pre-trained RoBERTa model with task-specific classification layers for fine-tuning. The number of iterations was set to 5. All deep learning models were implemented based on the deep learning framework PyTorch. The learning rate was set to 3e-5, the batch size was 256, the optimizer was AdamW, and Dropout was used as the regularization method to prevent overfitting.

Images