Method for converting natural language text into structured query language

Abstract

The present invention provides a method for converting natural language text into structured query language, which relates to the technical field of natural language processing. The method comprises: using a mutual information regularizer to randomly mask an input sequence to obtain local perturbation features, and maximizing the mutual information between the local perturbation features and the global features; using a learning pattern linking module to annotate a pattern linking data set, and simultaneously learning the embedding of tables and columns based on a Transformer model; constructing a Text‑to‑SQL model based on the mutual information regularizer and the learning pattern linking module, the Text‑to‑SQL model comprises a first-stage learning task and a second-stage learning task; the first-stage learning task comprises: using a pattern linking encoder to obtain a reordered question‑pattern sequence; the second-stage learning task comprises: using an encoder and a decoder to perform SQL parsing on the question‑pattern sequence. The method of the present invention can effectively solve the sequence perturbation attack problem and the question‑pattern linking problem faced by the current large model.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method for converting natural language text into structured query language. Background Art

[0002] SQL is a standard database query language, namely structured query language, but the difficulty in writing it hinders non-professional users from using it. With the development of artificial intelligence technology and autonomous database systems, people's requirements for natural language queries are getting higher and higher. It is no longer satisfied with simple data retrieval, but can quickly and effectively respond to complex query retrieval requirements involving logic and semantics. Text-to-SQL (generating SQL based on text, that is, converting natural language text into structured query language) is an important subtask in semantic parsing. It automatically generates corresponding SQL queries based on the user's natural language query. Helping non-professional users to quickly use database queries, this task currently has many application scenarios, such as intelligent customer service, question-answering systems, and search engines. Therefore, related research has attracted widespread attention from industry and academia. Given a natural language query and a corresponding database schema, the goal of the Text-to-SQL model is to generate an SQL query statement.

[0003] Text-to-SQL methods can be divided into the following categories: (1) Sequence encoder-based methods; in this method, the input is a sequence that connects a natural language query with a serialized database schema. Each token in the sequence is then encoded by a pre-trained language model encoder, such as encoder T5. The PICARD algorithm (an iterative algorithm) designs a constrained decoding process algorithm to detect and reject incorrect tokens that appear in the directed search phase. This improves the accuracy of SQL decoding. S2SQL and LGESQL (both existing Text-to-SQL methods) further integrate other structured knowledge tasks into T5 and use semantic enhancement to improve the performance of Text-to-SQL. (2) Graph encoder-based methods; in this method, the input is one or more heterogeneous graphs, where nodes represent natural language query tokens, tables, or columns, and edges represent the relationship between two nodes. For example, Graphix-T5 (a hybrid training model) uses a graph-aware network to enhance Text-to-SQL parsing, which introduces structured inductive biases into the decoder of Text-to-SQL, thereby improving the model's multi-hop reasoning ability. (3) Methods based on large models: SQL-PaLM (a model based on a large language model) proposes a Text-to-SQL model based on PaLM (a generative pre-trained model), designs a prompting method based on self-consistency, and fine-tunes LLMs (large language models). DIN-SQL (a model for natural language to SQL conversion) is based on LLMs and follows the prompts from least to most, breaking down complex Text-to-SQL tasks into several fixed subtasks (such as pattern linking, query classification, and decomposition), improving the parsing performance of LLMs on Text-to-SQL tasks. ChatGPT, CodeX, and GPT4 (all existing large-scale language models) generate SQL statements based on the prompts of LLMs, and have powerful Text-to-SQL generation capabilities in zero-shot scenarios.

[0004] Based on the existing technology, many Text-to-SQL models based on large language models encode natural language queries and database patterns and decode the corresponding SQL, achieving remarkable results on existing public benchmarks such as the Spider dataset. However, these results are obtained when the test data has the same distribution as the training data. This setting prevents the evaluation of the robustness of the model, especially when the data contains spurious patterns that do not exist in the actual environment. As a result, the performance of these methods on robust Text-to-SQL datasets (such as Spider-Syn, Dr.Spider dataset) has dropped significantly. We observe that there are two main problems with current models.

[0005] (1) Sequence perturbation attack problem. The input of current models is usually a question-pattern sequence. Therefore, perturbing the input sequence is the main attack threat faced by the model, and existing methods are difficult to effectively defend against such attacks. State-of-the-art models are also vulnerable to perturbations. Natural language question perturbations mainly replace important words in a sentence. For example, the model confuses the name of the selected column winner with the age of the winner and gives a paraphrased natural language query in which "Who" is used to imply the selected column. Database pattern perturbations replace tables and columns. For example, replacing the column name winnername with champ-name causes the model to miss the intent of "3 youngest winners". In addition, SQL statements can also be perturbed. For example, a simple change to the number of returned items (from LIMIT3 to LIMIT8) makes the model unable to detect the correct intent.

[0006] (2) The problem of question-pattern connection. Current research relies heavily on the precise lexical matching between natural language queries and assumptions about database schemas. However, natural language expressions are diverse, and there are different forms of expression for the same entity. For example, "company" can be expressed as both "manufacturer" and "supplier", and "debt ratio" and "leverage ratio" are the same concept. On the other hand, there may be similar table and column names in the database, causing link ambiguity. For example, "model" can refer to "car" model or "engine" model. In addition, when the model training is tested on an unseen database, out-of-domain words and phrases are difficult to understand. For example, "find comedy movies released in 2018" and "find students enrolled in 2018" actually need to be treated differently. Summary of the invention

[0007] The present invention aims to solve at least one of the technical problems existing in the prior art, namely, the sequence perturbation attack problem and the problem-pattern connection problem.

[0008] To this end, the present invention provides a method for converting natural language text into structured query language.

[0009] The present invention provides a method for converting natural language text into structured query language, comprising:

[0010] The mutual information regularizer is used to randomly mask the input sequence to obtain local perturbation features and maximize the mutual information between the local perturbation features and the global features.

[0011] We use a learned pattern linking module to annotate the pattern linking dataset and learn table and column embeddings simultaneously based on the Transformer model.

[0012] A Text-to-SQL model is constructed based on a mutual information regularizer and a learning pattern chaining module. The Text-to-SQL model includes a first-stage learning task and a second-stage learning task. The first-stage learning task includes: using a pattern chaining encoder to obtain a reordered question-pattern sequence; the second-stage learning task includes: using a Text-to-SQL encoder and a Text-to-SQL decoder to perform SQL parsing on the question-pattern sequence.

[0013] The method for converting natural language text into structured query language according to the above technical solution of the present invention may also have the following additional technical features:

[0014] In the above technical solution, the mutual information regularizer includes a random mask module, which is used to partially sample samples and randomly perform mask operations on the samples in each training iteration;

[0015] Among them, the smoothing model is used to replace the basic model.

[0016] In the above technical solution, the step of replacing the basic model with a smoothing model includes:

[0017] A smoothing classifier is used to replace a given base classifier, wherein the base classifier is used to classify randomly masked words;

[0018] The expression of the base classifier is:

[0019]

[0020] The expression of the smoothing classifier is:

[0021]

[0022] in, represents a given base classifier; Represents the masked input; represents a smoothing classifier; represents the input sample; argmax represents the average normalization function; Indicates category; Represents the label space; Represents the probability of the model's prediction results; represents the masked sample space; Indicates a set of indicators indicating whether Uniform sampling in the middle; express Length, express The number of unmasked words, given by [ ] calculated, Indicates the percentage of words that can be masked. Represents a text random mask operation;

[0023] use It means that after random masking, Return to Class The amount of information is:

[0024] .

[0025] In the above technical solution, maximizing the mutual information between the local perturbation feature and the global feature includes:

[0026] By maximizing the local perturbation features and the final The mutual information of , so that information can be transmitted normally under disturbance attacks, including:

[0027]

[0028] in, represents the maximization function, represents the amount of information of normal samples, Indicates the amount of sample information that is blocked; Represents the probability of the model prediction result based on the input sample x; represents the probability of the prediction result based on the label space y model, represents the joint probability.

[0029] In the above technical solution, when estimating the amount of information, the mutual information is not directly estimated, and a variational approximation method is used to obtain the lower bound of the mutual information as an approximation; wherein the InfoNCE loss function is used to estimate the amount of information, including:

[0030]

[0031] in, represents X and InfoNCE loss; represents the variational lower bound function; represents the critical function approximated by the neural network; X represents the input sequence; Represents the joint distribution from local perturbation features and global representation The positive samples extracted from From disturbances and useless features The negative samples extracted from the distribution of , i represents the sample number.

[0032] In the above technical solution, a learning-based pattern linking module is used to annotate the pattern linking dataset, and the embedding of tables and columns is learned simultaneously based on the Transformer model, including:

[0033] Schema linking is defined as a classification task where the learning objective is to classify patterns based on natural language queries. and database schema Identify pattern links ; where Q represents a natural language query set, represents a natural language query, Indicates the length of the natural language query set, Represents a collection of tables, Represents a collection of columns, Table schema link representing predictions, Column pattern links representing predictions;

[0034] Supervised learning is used to enable the model to learn the potential pattern link relationship between natural language queries and patterns;

[0035] The learning pattern linking module includes link annotations, a representation layer and a dense layer.

[0036] In the above technical solution, the link annotation is used to define the matching rules of the labels before training the learning mode link module; wherein, the natural language query set Q and the database mode The link relationship is matched, including:

[0037] Given a natural language query set and the corresponding database schema ,in, , , t represents a table, Represents a field. Indicates the number of tables; Indicates the number of fields;

[0038] Represent the input as The goal is to Annotation Mode Link Tags ,in, express The table in the table is the set of tags linked by the question. =1 means table T is linked by natural language query, otherwise 0; express The fields in the are the collection of tags to which the question is linked, =1 indicates that the column is linked via a natural language query, otherwise 0.

[0039] In the above technical solution, the representation layer is used to simultaneously learn table and column embedding based on the Transformer model, including:

[0040] Given an input sequence , the goal of the representation layer is to obtain the fitting label Table embedding and the corresponding column embedding ,in, represents the hidden layer size, Indicates the number of columns, represents all real numbers, Represents the table corresponding to the column, i represents the subscript of the column; Represented as an input sequence, it is defined as follows:

[0041]

[0042] Among them, | represents a separator, represents the input sequence, [CLS] [SEP] represents the sample separator, q i represents the i-th question, t represents the table, c represents the column, Indicates the number of columns, Indicates the number of tables;

[0043] The input sequence is encoded by the Transformer block, where the encoding core is weighted attention The calculation of the query matrix is ​​obtained by three matrices , the bond matrix Sum Matrix , defined as follows:

[0044]

[0045] Where m represents the number of query vectors, n represents the number of key vectors or value vectors, represents the dimension of the query matrix or key matrix, represents the dimension of the value matrix, represents the normalized exponential function;

[0046] According to the context information of natural language query and database mode at different positions, the input sequence Assign different weights to each word in ; connect the attention outputs of each head in series to get ,include:

[0047]

[0048]

[0049]

[0050]

[0051] in, Represents the multi-head attention mechanism function; Indicates the size of the head; Represents the column subscript and serves as , , The index,maps the input to query,key-value pairs; express function; represents a feed-forward neural network; Represents the output of multi-head attention; Indicates the first head attention; Indicates a connection operation; The weight matrix representing the attention; Represents single head attention; , , Represent the weights of matrix Q, matrix K and matrix V respectively.

[0052] The bidirectional long short-term memory network BiLSTM captures the long-term dependencies between tables and columns in the sequence, captures global features, and combines them with attention to enable the model to perform hierarchical feature extraction at different levels, including:

[0053]

[0054]

[0055] in, represents the embedding of the table, represents the embedding of the column, represents a two-layer bidirectional long short-term memory network, represents a nonlinear fully connected layer;

[0056] Inject the column embeddings into the corresponding table embeddings to get the final table embeddings, including:

[0057]

[0058]

[0059] in, Represents the final table embedding.

[0060] In the above technical solution, the dense layer is used to map the table embedding and column embedding to the output result, including:

[0061] Create two different MLP modules to convert the embedded features into one-dimensional vectors, and use these one-dimensional vectors for the final classification task, including:

[0062]

[0063]

[0064] in, and Represent the probability of table and column prediction respectively; represents the normalization function; , represents the learning parameters of the table; , represents the learning weights of the table, , represents the learning parameters of the column, , Represents the learning weight of the column. The above parameters are all trainable parameters in the MLP module;

[0065] Use focal loss as the loss function of the learning mode link module, including:

[0066]

[0067] in, represents the loss function of the learning mode link module, represents the focal loss function, Indicates the A real label, Indicates the first A real label.

[0068] In the above technical solution, the step of using a pattern chaining encoder to obtain a reordered question-pattern sequence includes:

[0069] make Represents a non-overlapping span sequence of the problem, initializes the learning pattern link module to obtain a pattern link set ,include:

[0070]

[0071] in, represents the learning mode linking module;

[0072] According to the input sequence Linking sets with patterns to get sorted input sequences :

[0073]

[0074] in, Represents a sorting algorithm;

[0075] The method of using a Text-to-SQL encoder and a Text-to-SQL decoder to perform SQL parsing on a question-pattern sequence includes:

[0076] Using the instruction-based fine-tuning text-to-text transfer transformer language model Flan-T5 in the Text-to-SQL encoder for sorted input sequences Encoding includes:

[0077]

[0078] in, represents instruction-based fine-tuning, represents the text-to-text transfer transformer language model, Indicates the encoding result;

[0079] Use the MIR module to add regularization to the TS encoder, including:

[0080]

[0081] in, represents the mutual information regularizer; Represents the encoding result after adding regularization;

[0082] In the Text-to-SQL decoder, the frame is parsed first, and then the SQL query is parsed; wherein, a sequence-to-sequence model seq2seq is used for parsing, and the objective function of the sequence-to-sequence model seq2seq is for:

[0083]

[0084] in, Indicates the number of text-to-SQL instances. represents the input order of the i-th instance, Represents a sequence problem, an ordering pattern sequence, and optional foreign key relationships, Represents the SQL keywords extracted from the skeleton l. Represents the predicted probability.

[0085] In summary, due to the adoption of the above technical features, the beneficial effects of the present invention are:

[0086] The present invention designs a novel Text-to-SQL architecture (InfoSQL for short) with provable robustness from a theoretical perspective; the input sequence is actively randomly masked to obtain local perturbation features, and then the mutual information between the local perturbation features and the global features is maximized.

[0087] In addition, the present invention also proposes a learning mode linking module (TSL), annotates a new mode linking dataset, and simultaneously learns table and column embedding based on Transformer to improve the model's problem-mode alignment in robust scenarios. It can effectively predict cross-domain patterns and improve the problem-mode alignment in robust scenarios, further improving robustness.

[0088] InfoSQL achieves superior robustness scores on four datasets: robust Text-to-SQL benchmarks Spider-DK, Spider-Syn, Spider-Realistic, and Dr.Spider.

[0089] Additional aspects and advantages of the present invention will become apparent from the following description or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0091] Figure 1 The present invention is a schematic diagram of pattern links in a method for converting natural language text into structured query language according to an embodiment of the present invention. DETAILED DESCRIPTION

[0092] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0093] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0094] Refer to the following Figure 1 The method for converting natural language text into structured query language provided according to some embodiments of the present invention is described.

[0095] Some embodiments of the present application provide a method for converting natural language text into structured query language.

[0096] The first embodiment of the present invention proposes a method for converting a natural language text into a structured query language, comprising:

[0097] The mutual information regularizer is used to randomly mask the input sequence to obtain local perturbation features and maximize the mutual information between the local perturbation features and the global features.

[0098] We use a learned pattern linking module to annotate the pattern linking dataset and learn table and column embeddings simultaneously based on the Transformer model.

[0099] A Text-to-SQL model is constructed based on a mutual information regularizer and a learning pattern chaining module. The Text-to-SQL model includes a first-stage learning task and a second-stage learning task. The first-stage learning task includes: using a pattern chaining encoder to obtain a reordered question-pattern sequence; the second-stage learning task includes: using a Text-to-SQL encoder and a Text-to-SQL decoder to perform SQL parsing on the question-pattern sequence.

[0100] In the above method, the mutual information regularizer is used to solve the sequence perturbation attack problem, and the learning pattern linking module is used to solve the problem-pattern connection problem. Finally, a neural model of the text-to-SQL parser disclosed in the present invention, namely the InfoSQL model, is given.

[0101] Based on the mutual information regularizer disclosed in the present invention, the robustness of the output results of the Text-to-SQL model can be improved, and the robustness can also be proven.

[0102] When facing a perturbation attack, important key words and phrases used by the model to generate SQL can be easily perturbed, thereby interfering with the model's output, such as when an attacker replaces questions, patterns, or pattern links. In some embodiments, in order to improve the defense capability of the model, in each training iteration, a small batch of samples is first sampled and randomly masked.

[0103] Specifically, the mutual information regularizer includes a random mask module, which is used to partially sample samples and randomly perform mask operations on the samples in each training iteration;

[0104] In this case, a smoother model is used to replace the base model. , which is easier to verify by integrating the output with some random masked inputs.

[0105] In a specific embodiment, the step of replacing the basic model with a smoothing model includes:

[0106] A smoothing classifier is used to replace a given base classifier, wherein the base classifier is trained to classify randomly masked words;

[0107] The expression of the base classifier is:

[0108]

[0109] The expression of the smoothing classifier is:

[0110]

[0111] in, represents a given base classifier; Represents the masked input; represents a smoothing classifier; represents the input sample; argmax represents the average normalization function; Indicates category; Represents the label space; Represents the probability of the model's prediction results; represents the masked sample space; Indicates a set of indicators indicating whether Uniform sampling in the middle; express Length, express The number of unmasked words, given by [ ] calculated, Indicates the percentage of words that can be masked. Represents a text random mask operation;

[0112] use It means that after random masking, Return to Class The amount of information is:

[0113] .

[0114] The robustness of the above smoothing classifier has the following theorem:

[0115] Given a text input and , Applicable to all categories ,have:

[0116]

[0117] in, represents the amount of information of normal samples, Indicates the amount of sample information that is blocked, is a constant, is a constant, is the output probability of the basic classifier;

[0118]

[0119]

[0120] in, is the perturbed sample, It is a normal sample.

[0121] The proof of the above theorem is as follows:

[0122]

[0123]

[0124] if ,Then and All subscripts are the same .in this case, = ,but:

[0125]

[0126] According to the above formula and the probability of non-negativity, and Subtracting them gives:

[0127]

[0128] According to the output probability of the basic classifier The definition of is:

[0129]

[0130] And then:

[0131]

[0132] Notice:

[0133]

[0134] The last equation is It is a unified choice Elements , here are Of all these choices, there are Does not contain any elements from Constraints ,but:

[0135]

[0136] Since the amount of information is difficult to calculate, we assume for The lower bound of To constrain Specifically, the random construction Input Then, Class count Assumptions If it follows the binomial distribution parameter:

[0137]

[0138] in express - quantile of beta distribution with parameter, , and is the confidence level.

[0139] According to the above theorem, we can draw the following inferences:

[0140] For text , , ,if:

[0141]

[0142] Then, according to the confidence have:

[0143]

[0144] Proof: According to probability We have:

[0145]

[0146] For the last inequality in the above theorem , we can get:

[0147] if ( The true label in ) and if Smoothing Classifier At the input Proven to be robust.

[0148] After completing the random mask, the local perturbation feature is maximized with the final one put into the classification head. The mutual information of tokens enables information to be transmitted normally under disturbance attacks.

[0149] In some embodiments, maximizing the mutual information between the local perturbation feature and the global feature includes:

[0150] By maximizing the local perturbation features and the final The mutual information of , so that information can be transmitted normally under disturbance attacks, including:

[0151]

[0152] in, represents the maximization function, represents the amount of information of normal samples, Indicates the amount of sample information that is blocked; Represents the probability of the model prediction result based on the input sample x; represents the probability of the prediction result based on the label space y model, represents the joint probability.

[0153] Due to the difficulty of solving mutual information, it is not necessary to estimate the mutual information directly. In some embodiments, when estimating the amount of information, a variational approximation method is used to obtain the lower bound of the mutual information as an approximation; wherein the InfoNCE loss function is used to estimate the amount of information, including:

[0154]

[0155] in, represents X and InfoNCE loss; represents the variational lower bound function; represents the critical function approximated by the neural network; X represents the input sequence; Represents the joint distribution from local perturbation features and global representation The positive samples extracted from Represents disturbances and useless features The negative samples extracted from the distribution of , i represents the sample number.

[0156] In some embodiments, in the pattern linking learning part, a learning pattern linking module (TSL for short) is used to annotate the pattern linking dataset, and the embedding of tables and columns is simultaneously learned based on the Transformer model, specifically including:

[0157] We define pattern linking as a classification task where the learning objective is to obtain and database schema Identify pattern links ; where Q represents a natural language query set, represents a natural language query, Indicates the length of the natural language query set, Represents a collection of tables, Represents a collection of columns, Table schema link representing predictions, Column pattern link representing the prediction.

[0158] According to the potential pattern links in NLQ, they should be linked to a table or column; however, it is difficult to identify these links with existing methods; supervised learning is used in the present disclosure to enable the model to learn the potential pattern link relationship between natural language queries and patterns, and ensure generalization ability under new data.

[0159] The learning pattern link module includes link annotation, representation layer and dense layer; specifically, the pattern link is as follows: Figure 1 shown.

[0160] Specifically, the link annotation is used to define the label matching rules before training the learning model link module. Matching words with SQL statements to get labels, ignoring the natural language query set in NLQ Different from the existing methods, the present invention relies on natural language query set Q and database schema In a specific embodiment, the Spider dataset is annotated with pattern links to establish a supervised learning task, which specifically includes:

[0161] Given a natural language query set and the corresponding database schema ,in, , , t represents a table, Represents a field. Indicates the number of tables; Indicates the number of fields;

[0162] Represent the input as The goal is to Annotation Mode Link Tags ,in, express The table in the table is the set of tags linked by the question. =1 means table T is linked by natural language query, otherwise 0; express The fields in the are the collection of tags to which the question is linked, =1 indicates that the column is linked via a natural language query, otherwise 0.

[0163] Table 1 gives all the labels and their matching rule examples used in this work. In Table 1, the label column indicates the type of pattern link, which is divided into table and field, the question column indicates NLQ and the pattern column indicates that the database pattern consists of table name and column name. The labeled words in the question column indicate that they are linked to the table or column in the pattern column. Usually, the values ​​in SQL are related to the fields, so the values ​​appearing in the question are also linked to the fields. In addition, when the columns are linked, the tables should also be linked. For example, the word "older" in "How many heads of the departments are older than 56?" is linked to the column "head.older", so it can be further inferred that the head table is also linked to the question.

[0164] Table 1 Proposed pattern linking dataset

[0165]

[0166] In some embodiments, the representation layer is used to simultaneously learn table and column embeddings based on a Transformer model, including:

[0167] Given an input sequence , the goal of the representation layer is to obtain the fitting label Table embedding and the corresponding column embedding ,in, represents the hidden layer size, Indicates the number of columns, represents all real numbers, Indicates the table corresponding to the column, Indicates the column subscript; Represented as an input sequence, it is defined as follows:

[0168]

[0169] Among them, | represents a separator, represents the input sequence, [CLS] [SEP] represents the sample separator, q i represents the i-th question, t represents the table, c represents the column, Indicates the number of columns, Indicates the number of tables;

[0170] The input sequence is encoded by the Transformer block, where the encoding core is weighted attention The calculation of the query matrix is ​​obtained by three matrices , the bond matrix Sum Matrix , defined as follows:

[0171]

[0172] Where m represents the number of query vectors, n represents the number of key vectors or value vectors, represents the dimension of the query matrix or key matrix, represents the dimension of the value matrix, represents the normalized exponential function;

[0173] According to the context information of natural language query and database mode at different positions, the input sequence Assign different weights to each word in ; connect the attention outputs of each head in series to get ,include:

[0174]

[0175]

[0176]

[0177]

[0178] in, Represents the multi-head attention mechanism function; Indicates the size of the head; Represents the column subscript and serves as , , The index,maps the input to query,key-value pairs; express function; represents a feed-forward neural network; Represents the output of multi-head attention; Indicates the first head attention; Indicates a connection operation; The weight matrix representing the attention; Represents single head attention; , , Represent the weights of matrix Q, matrix K and matrix V respectively;

[0179] Although attention contains rich semantic information, some hierarchical features are not included. In a specific embodiment, a bidirectional long short-term memory network BiLSTM is used to capture the long-term dependencies between tables and columns in the sequence, capture global features, and combine with attention to enable the model to perform hierarchical feature extraction at different levels and more comprehensively understand the link relationship between NLQ and patterns, including:

[0180]

[0181]

[0182] in, represents the embedding of the table, represents the embedding of the column, represents a two-layer bidirectional long short-term memory network, represents a nonlinear fully connected layer;

[0183] Usually, in the process of linking tables, some NLQs only mention the column names in the table instead of the complete table names corresponding to the columns. The missing table names will make it difficult for the model to link to the correct table names. Therefore, the column embeddings are injected into the corresponding table embeddings to obtain the final table embeddings, including:

[0184]

[0185]

[0186] in, Represents the final table embedding.

[0187] In some embodiments, the dense layer is used to map the table embedding and the column embedding to the output result, including:

[0188] Create two different MLP modules (multi-layer perceptron modules) to convert high-dimensional embedding features into one-dimensional vectors, and use these one-dimensional vectors for the final classification task, so that the model can learn appropriate semantic representations and classification decisions on the training data, including:

[0189]

[0190]

[0191] in, and Represent the probability of table and column prediction respectively; represents the normalization function; , represents the learning parameters of the table; , represents the learning weights of the table, , represents the learning parameters of the column, , Represents the learning weight of the column. The above parameters are all trainable parameters in the MLP module;

[0192] It has been observed that the label distribution in the training set is highly unbalanced, with more negative examples than positive examples. This is because in a Text-to-SQL sample, usually only a small number of tables and columns will be linked by NLQ. Therefore, in some embodiments, focus loss can be used as the loss function of the learning pattern linking module. Specifically, during the training process, the focus loss function can give a small number of target categories and misclassified samples a larger weight, thereby increasing the model's attention to these samples. The loss of TSL consists of table classification loss and column classification loss. The loss function includes:

[0193]

[0194] in, represents the loss function of the learning mode link module, represents the focal loss function, Indicates the A real label, Indicates the first A real label.

[0195] A Text-to-SQL model (InfoSQL) is created based on the mutual information regularizer and TSL module proposed in the above embodiment, which accepts a question, a database schema and schema linking results as input.

[0196] In the first stage learning task, the pattern link encoder is used to obtain a reordered question-pattern sequence, including:

[0197] make Represents a non-overlapping span sequence of the problem, initializes the learning pattern link module to obtain a pattern link set ,include:

[0198]

[0199] in, represents the learning mode linking module;

[0200] Due to the limitation of LLMs on the length of input sequences, the model can usually only process questions and pattern sequences with a length of 512 tokens (the smallest unit of input data used by the model in natural language processing tasks, which can be words, letters, phrases or characters, etc.), which will cause the key tables or fields used for SQL generation to be lost. The order of the tables and columns linked by the question in the sequence is improved by reordering. Linking sets with patterns to get sorted input sequences :

[0201]

[0202] in, Represents a sorting algorithm;

[0203] In the second stage learning task, the question-pattern sequence is parsed using the Text-to-SQL encoder and the Text-to-SQL decoder, including:

[0204] The encoder is used to encode the sorted question-pattern sequence to obtain an embedding with semantic representation. Specifically, the text-to-text transfer transformer language model Flan-T5 based on instruction fine-tuning is used in the Text-to-SQL encoder to encode the sorted input sequence Encoding includes:

[0205]

[0206] in, represents instruction-based fine-tuning, represents the text-to-text transfer transformer language model, Indicates the encoding result;

[0207] The MIR module is used to add regularization to the TS encoder to improve the mutual information of local and global embeddings and enhance the robustness of the model, including:

[0208]

[0209] in, represents the mutual information regularizer; Represents the encoding result after adding regularization;

[0210] The decoder is used to parse the embedding to obtain the SQL prediction result. Specifically, in the Text-to-SQL decoder, the frame is parsed first, and then the SQL query is parsed. The process of SQL generation will become simpler at each decoding stage; this is because the decoder can choose to copy "data" from the input sequence or copy SQL keywords from the previously parsed frame. Among them, the sequence-to-sequence model seq2seq is used for parsing, and the objective function of the sequence-to-sequence model seq2seq is for:

[0211]

[0212] in, Indicates the number of text-to-SQL instances. represents the input order of the i-th instance, Represents a sequence problem, an ordering pattern sequence, and optional foreign key relationships, Indicates the SQL keywords extracted from the SQL skeleton l. Indicates the predicted probability. The SQL skeleton l is a professional term in the database.

[0213] In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0214] Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for converting natural language text into structured query language, characterized in that: include: The mutual information regularizer is used to randomly mask the input sequence to obtain local perturbation features and maximize the mutual information between the local perturbation features and the global features. We use a learned pattern linking module to annotate the pattern linking dataset and learn table and column embeddings simultaneously based on the Transformer model. Building a Text-to-SQL model based on a mutual information regularizer and a learning pattern linking module, wherein the Text-to-SQL model includes a first-stage learning task and a second-stage learning task; The first stage learning task includes: using a pattern linking encoder to obtain a reordered question-pattern sequence; the second stage learning task includes: using a Text-to-SQL encoder and a Text-to-SQL decoder to perform SQL parsing on the question-pattern sequence; The mutual information regularizer includes a random mask module, which is used to partially sample samples and randomly mask the samples in each training iteration; Among them, the smoothing model is used to replace the basic model; The maximizing the mutual information between the local perturbation feature and the global feature includes: By maximizing the mutual information between the local perturbation features and the global features finally put into the classification head, information can be transmitted normally under perturbation attacks, including: in, represents the maximization function, represents the amount of information of normal samples, Indicates the amount of sample information that is blocked; Represents the probability of the model prediction result based on the input sample x; represents the probability of the prediction result based on the label space y model, represents the joint probability; The learning mode linking module includes a link annotation, a representation layer and a dense layer; the link annotation is used to define the matching rules of the labels before training the learning mode linking module; the representation layer is used to simultaneously learn the embedding of the table and the column based on the Transformer model; the dense layer is used to map the table embedding and the column embedding to the output result; The method of using a pattern linking encoder to obtain a reordered question-pattern sequence includes: make Represents a non-overlapping span sequence of the problem, initializes the learning pattern link module to obtain a pattern link set ,include: in, represents the learning mode linking module; According to the input sequence Linking sets with patterns to get sorted input sequences : in, Represents a sorting algorithm; The method of using a Text-to-SQL encoder and a Text-to-SQL decoder to perform SQL parsing on a question-pattern sequence includes: Using the instruction-based fine-tuning text-to-text transfer transformer language model Flan-T5 in the Text-to-SQL encoder for sorted input sequences Encoding includes: in, represents instruction-based fine-tuning, represents the text-to-text transfer transformer language model, Indicates the encoding result; Use the MIR module to add regularization to the TS encoder, including: in, represents the mutual information regularizer; Represents the encoding result after adding regularization; In the Text-to-SQL decoder, the frame is parsed first, and then the SQL query is parsed; wherein, a sequence-to-sequence model seq2seq is used for parsing, and the objective function of the sequence-to-sequence model seq2seq is for: in, Indicates the number of text-to-SQL instances. represents the input order of the i-th instance, Represents a sequence problem, an ordering pattern sequence, and optional foreign key relationships, Indicates the SQL keywords extracted from the SQL skeleton l. Represents the predicted probability.

2. The method for converting natural language text into structured query language according to claim 1, characterized in that: The step of replacing the basic model with a smoothing model comprises: A smoothing classifier is used to replace a given base classifier, wherein the base classifier is used to classify randomly masked words; The expression of the base classifier is: The expression of the smoothing classifier is: in, represents a given base classifier; Represents the masked input; represents a smoothing classifier; represents the input sample; argmax represents the average normalization function; Indicates category; Represents the label space; Represents the probability of the model's prediction results; represents the masked sample space; Indicates a set of indicators indicating whether Uniform sampling in the middle; express Length; express The number of unmasked words, given by [ ] calculated, Indicates the percentage of words that can be masked. Represents a text random mask operation; use It means that after random masking, Return to Class The amount of information is: 。 3. The method for converting natural language text into structured query language according to claim 2, characterized in that: When estimating the amount of information, the mutual information is not estimated directly, and the variational approximation method is used to obtain the lower bound of the mutual information as an approximation; among them, the InfoNCE loss function is used to estimate the amount of information, including: in, represents X and InfoNCE loss; represents the variational lower bound function; represents the critical function approximated by the neural network; X represents the input sequence; Represents the joint distribution from local perturbation features and global representation The positive samples extracted from Represents disturbances and useless features The negative samples extracted from the distribution of , i represents the sample number.

4. The method for converting natural language text into structured query language according to claim 1, characterized in that: The learned pattern linking module is used to annotate the pattern linking dataset and learn table and column embeddings simultaneously based on the Transformer model, including: Schema linking is defined as a classification task where the learning objective is to classify patterns based on natural language queries. and database schema Identify pattern links ; where Q represents a natural language query set, represents a natural language query, Indicates the length of the natural language query set, Represents a collection of tables, Represents a collection of columns, Table schema link representing predictions, Column pattern links representing predictions; Supervised learning is used to enable the model to learn the potential pattern link relationship between natural language queries and patterns.

5. The method for converting natural language text into structured query language according to claim 4, characterized in that: Before training the learning mode link module, the label matching rules are defined, including: Rely on natural language to query the set Q and the database schema Match the link relationship; Given a natural language query set and the corresponding database schema ,in, , , t represents a table, Represents a field. Indicates the number of tables; Indicates the number of fields; Represent the input as The goal is to Annotation Mode Link Tags ,in, express The table in the table is the set of tags linked by the question. =1 means table T is linked by natural language query, otherwise 0; express The fields in the are the collection of tags to which the question is linked, =1 indicates that the column is linked via a natural language query, otherwise 0.

6. The method for converting natural language text into structured query language according to claim 5, characterized in that: The Transformer model-based method of simultaneously learning table and column embeddings includes: Given an input sequence , the goal of the representation layer is to obtain the fitting label Table embedding and the corresponding column embedding ,in, represents the hidden layer size, Indicates the number of columns, represents all real numbers, Represents the table corresponding to the column, i represents the subscript of the column; Represented as an input sequence, it is defined as follows: Among them, | represents a separator, represents the input sequence, [CLS] [SEP] represents the sample separator, q i represents the i-th question, t represents the table, c represents the column, Indicates the number of columns, Indicates the number of tables; The input sequence is encoded by the Transformer block, where the encoding core is weighted attention The calculation of the query matrix is ​​obtained by three matrices , the bond matrix Sum Matrix , defined as follows: Where m represents the number of query vectors, n represents the number of key vectors or value vectors, represents the dimension of the query matrix or key matrix, represents the dimension of the value matrix, represents the normalized exponential function; According to the context information of natural language query and database mode at different positions, the input sequence Assign different weights to each word in ; connect the attention outputs of each head in series to get ,include: in, Represents the multi-head attention mechanism function; Indicates the size of the head; Represents the column subscript and serves as , , The index,maps the input to query,key-value pairs; express function; represents a feed-forward neural network; Represents the output of multi-head attention; Indicates the first head attention; Indicates a connection operation; The weight matrix representing the attention; Represents single head attention; , , Represent the weights of matrix Q, matrix K and matrix V respectively; The bidirectional long short-term memory network BiLSTM captures the long-term dependencies between tables and columns in the sequence, captures global features, and combines them with attention to enable the model to perform hierarchical feature extraction at different levels, including: in, represents the embedding of the table, represents the embedding of the column, represents a two-layer bidirectional long short-term memory network, represents a nonlinear fully connected layer; Inject the column embeddings into the corresponding table embeddings to get the final table embeddings, including: in, Represents the final table embedding.

7. The method for converting natural language text into structured query language according to claim 6, characterized in that: The mapping of table embedding and column embedding to output results includes: Create two different MLP modules to convert the embedded features into one-dimensional vectors, and use these one-dimensional vectors for the final classification task, including: in, and Represent the probability of table and column prediction respectively; represents the normalization function; , represents the learning parameters of the table; , represents the learning weights of the table, , represents the learning parameters of the column, , Represents the learning weight of the column. The above parameters are all trainable parameters in the MLP module; Use focal loss as the loss function of the learning mode link module, including: in, represents the loss function of the learning mode link module, represents the focal loss function, Indicates the A real label, Indicates the first A real label.

Images