Devices and methods for generating an SQL query based on a natural language query

Abstract

A data processing apparatus (100) is disclosed for semantic parsing of a natural language query having a plurality of natural language query tokens for generating an SQL query for a relational database having a database schema with a plurality of database schema elements. The data processing apparatus (100) comprises an encoder (101c) configured to select, based on the natural language query and the database schema, for each of the plurality of natural language query tokens and each of the plurality of database schema elements an SQL Semantic Prediction, SSP, label of a plurality of SSP labels, wherein each SSP label is associated with at least one SQL clause. Moreover, the data processing apparatus (100) comprises a decoder configured to generate the SQL query based on the plurality of SQL clauses associated with the plurality of selected SSP labels. Furthermore, a corresponding computer-implemented data processing method is disclosed.

Description

[0001] Devices and methods for generating an SQL query based on a natural language query

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to information processing technology. More specifically, the present disclosure relates to devices and methods for generating an SQL query based on a natural language query.

[0004] BACKGROUND

[0005] While tabular relational data is a cornerstone of an increasing number of enterprises, it can mostly be accessed only by people familiar with specific query languages, such as Structured Query Language (SQL). Text-to-SQL semantic parsing is the task of transforming questions in a natural language, into SQL queries that can be executed by a database, in order for the appropriate results to be returned to the end-user. Automated systems capable of performing Text-to-SQL semantic parsing, i.e. transforming a natural language query into an SQL query can enable non-technical people to effortlessly access the data, without requiring assistance by SQL experts.

[0006] As a result, Text-to-SQL semantic parsing systems have become increasingly popular over the recent years. However, despite the substantial demand, conventional Text-to-SQL semantic parsing systems have notable limitations, such as working only for very simple queries and data sources and exhibiting slow processing times.

[0007] Another limitation of conventional Text-to-SQL semantic parsing systems is their reliance on the monolingual setting, where both the natural language questions and the database schema are in the same language. While schema information of non-English databases is often available in English, there are several challenges in transferring a monolingual system to a cross-lingual setup, where the natural language question is in a different language than the database schema. Most of the existing cross-lingual approaches focus on shared database semantic parsing setups where evaluation databases are known during training.

[0008] SUMMARY

[0009] It is an objective to provide improved devices and methods for generating an SQL query based on a natural language query, i.e. for Text-to-SQL semantic parsing. The foregoing and other objectives are achieved by the subject matter of the independent claims. Further implementation forms are apparent from the dependent claims, the description and the figures.

[0010] According to a first aspect, a data processing apparatus is provided for semantic parsing of a natural language query having a plurality of natural language query tokens for generating a structured query language, SQL, query for a relational database having a database schema with a plurality of database schema elements. The data processing apparatus comprises an encoder configured to select, based on the natural language query and the database schema, for each of the plurality of natural language query tokens and each of the plurality of database schema elements an SQL Semantic Prediction, SSP, label of a plurality of pre-defined SSP labels (also referred to as SSP classification labels), wherein each SSP label is associated with, i.e. corresponds to at least one SQL clause. Moreover, the data processing apparatus comprises a decoder configured to generate the SQL query based on the plurality of SQL clauses associated with the plurality of selected SSP labels. Thus, an improved data processing apparatus is provided for Text-to-SQL semantic parsing in a fast and scalable way.

[0011] In a further possible implementation form, the decoder comprises a predefined set of deterministic rules for generating the SQL query based on the plurality of SQL clauses associated with the plurality of selected SSP labels.

[0012] In a further possible implementation form, the encoder comprises a trainable encoder machine learning, ML, model, in particular, a trainable encoder neural network configured to select, based on the natural language query and the database schema, for each of the plurality of natural language query tokens and each of the plurality of database schema elements an SSP label of the plurality of pre-defined SSP labels.

[0013] In a further possible implementation form, the data processing apparatus is further configured to pre-train the encoder ML model based on a loss function, wherein the loss function comprises a loss function component defining a quality measure of the plurality of SSP labels based on the natural language query and the database schema of the relational database.

[0014] In a further possible implementation form, the data processing apparatus is further configured to determine, based on the natural language query, the natural language of the natural language query, wherein the loss function comprises a further loss function component defining a quality measure of the prediction of the natural language predicted based on the natural language query. In a further possible implementation form, the natural language query is expressed in a first natural language and in a second natural language, wherein the loss function comprises a further loss function component defining a quality measure of the encoder ML model to uncover masked tokens of the natural language query in the first and second natural language.

[0015] In a further possible implementation form, the data processing apparatus is further configured to train the encoder ML model based on a training set.

[0016] In a further possible implementation form, the data processing apparatus is further configured to: extract from an initial training set in a first natural language a plurality of natural language query templates and a plurality of corresponding SQL query templates; obtain for each of the plurality of natural language query templates in the first natural language a translation into a second natural language for obtaining a plurality of natural language query templates in the second natural language; generate the training set in the second natural language based on the plurality of natural language query templates in the second natural language and the plurality of corresponding SQL query templates.

[0017] In a further possible implementation form, for generating the training set in the second natural language based on the plurality of natural language query templates in the second natural language and the plurality of corresponding SQL query templates, the data processing apparatus is configured to sample at least one database in the second natural language including a plurality of table names, column names of tables and / or database elements of tables from the at least one database in the second natural language and to insert the plurality of table names, column names of tables and / or database elements of tables into the plurality of natural language query templates in the second natural language and the plurality of SQL query templates for generating a plurality of natural language queries in the second natural language by populating multiple instances of each natural language query template.

[0018] In a further possible implementation form, the data processing apparatus is further configured to: extract from an initial training set a plurality of natural language query templates and a plurality of corresponding SQL query templates; generate the training set based on the plurality of natural language query templates and the plurality of corresponding SQL query templates by sampling at least one database, in particular a plurality of table names, column names of tables and / or database elements of tables from the at least one database and to insert the plurality of table names, column names of tables and / or database elements of tables into the plurality of natural language query templates and the plurality of SQL query templates for generating a plurality of natural language queries by populating multiple instances of each natural language query template.

[0019] According to a second aspect a computer-implemented data processing method is provided for semantic parsing of a natural language query having a plurality of natural language query tokens for generating a structured query language, SQL, query for a relational database having a database schema with a plurality of database schema elements. The computer-implemented data processing method comprises the steps of: selecting, based on the natural language query and the database schema, for each of the plurality of natural language query tokens and each of the plurality of database schema elements an SQL Semantic Predictions, SSP, label of a plurality of pre-defined SSP labels, wherein each SSP label is associated with, i.e. corresponds to at least one SQL clause; and generating the SQL query based on the plurality of SQL clauses associated with the plurality of selected SSP labels.

[0020] The computer-implemented data processing method according to the second aspect of the present disclosure can be performed by the data processing apparatus according to the first aspect of the present disclosure. Thus, further features of the computer-implemented data processing method according to the second aspect of the present disclosure result directly from the functionality of the data processing apparatus according to the first aspect of the present disclosure as well as its different implementation forms described above and below.

[0021] According to a third aspect a computer program product is provided, comprising a computer- readable storage medium for storing program code which causes a computer or a processor to perform the computer-implemented data processing method according to the second aspect when the program code is executed by the computer or the processor.

[0022] Details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims.

[0023] BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In the following, embodiments of the present disclosure are described in more detail with reference to the attached figures and drawings, in which: Fig. 1 shows a schematic diagram illustrating a data processing apparatus according to an embodiment for generating an SQL query based on a natural language query;

[0025] Figs. 2a, b show schematic diagrams illustrating several data processing modules implemented by the data processing apparatus according to an embodiment for generating an SQL query based on a natural language query;

[0026] Figs. 3a-c illustrate two conventional decoding approaches and a decoding approach implemented by a data processing apparatus according to an embodiment for generating an SQL query based on a natural language query;

[0027] Fig. 4 illustrates an example for generating an SQL query based on a natural language query by a data processing apparatus according to an embodiment;

[0028] Fig. 5 schematically illustrates different processing stages implemented by a data processing apparatus according to an embodiment for generating based on a training data set in a first natural language a training data set in a second natural language; and

[0029] Fig. 6 is a flow diagram illustrating a computer-implemented data processing method according to an embodiment for generating an SQL query based on a natural language query and a relevant database schema.

[0030] In the following, identical reference signs refer to identical or at least functionally equivalent features.

[0031] DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] In the following description, reference is made to the accompanying figures, which form part of the disclosure, and which show, by way of illustration, specific aspects of embodiments of the present disclosure or specific aspects in which embodiments of the present disclosure may be used. It is understood that embodiments of the present disclosure may be used in other aspects and comprise structural or logical changes not depicted in the figures. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims.

[0033] For instance, it is to be understood that a disclosure in connection with a described method may also hold true for a corresponding device or system configured to perform the method and vice versa. For example, if one or a plurality of specific method steps are described, a corresponding device may include one or a plurality of units, e.g. functional units, to perform the described one or plurality of method steps (e.g. one unit performing the one or plurality of steps, or a plurality of units each performing one or more of the plurality of steps), even if such one or more units are not explicitly described or illustrated in the figures. On the other hand, for example, if a specific apparatus is described based on one or a plurality of units, e.g. functional units, a corresponding method may include one step to perform the functionality of the one or plurality of units (e.g. one step performing the functionality of the one or plurality of units, or a plurality of steps each performing the functionality of one or more of the plurality of units), even if such one or plurality of steps are not explicitly described or illustrated in the figures. Further, it is understood that the features of the various exemplary embodiments and / or aspects described herein may be combined with each other, unless specifically noted otherwise.

[0034] Figure 1 shows a schematic diagram illustrating a data processing apparatus 100 according to an embodiment for generating an SQL query based on a natural language query. In an embodiment, the data processing apparatus 100 may be implemented as a cloud server 100. As illustrated in Figure 1 , the data processing apparatus 100 may comprise processing circuitry 101 , a communication interface 103 and / or a memory 105. The processing circuitry 101 may be implemented in hardware and / or software and may comprise digital circuitry, or both analog and digital circuitry. Digital circuitry may comprise components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or general-purpose processors. The communication interface 103 may be configured to communicate with other network devices via wired and / or wireless connections. The memory 105 of the data processing apparatus 100 may be configured to store executable program code which, when executed by the processing circuitry 101 , causes the data processing apparatus 100 to perform the functions and methods described herein.

[0035] A user 110 may be using a user device 111a, b, in particular a user mobile phone 111a or any other computing device 111 b, to interact with the data processing apparatus100 via a communication network 120, for instance, the Internet 120. The data processing apparatus 100 may implement a search platform or a search mobile application. The natural language query 205 that is input by the user 110 is received by the data processing apparatus 100 for generating an SQL query 207 (as illustrated in more detail in figure 2a). In an embodiment, the data processing apparatus 100 may return the generated SQL query 207 to the user device 111a,b. Alternatively or additionally, the data processing apparatus 100 may provide the generated SQL query 207 to a database (which in an embodiment may be a component of the data processing apparatus 100) and return the results of the SQL query 207 to the user device 111 a,b, as illustrated in figure 2a.

[0036] As will be described in more detail under further reference to figures 2a and 2b, which show different data processing modules implemented by the data processing apparatus 100 according to an embodiment, the processing circuitry 101 of the data processing apparatus 100 is configured for semantic parsing of a natural language query 205 (provided, for instance, by the user 110) having a plurality of natural language query tokens for generating the SQL query 207 for a relational database 203 having a database schema 206 with a plurality of database schema elements. As illustrated in figure 2a and as will be described in more detail below, the data processing apparatus 100 comprises an encoder 101c (which may be implemented by the processing circuitry 101 of the data processing apparatus 100) configured to select, based on the natural language query 205 and the database schema 206, for each of the plurality of natural language query tokens and each of the plurality of database schema elements an SQL Semantic Prediction, SSP, label of a plurality of SSP labels, wherein each SSP label is associated with one or more SQL clauses. Moreover, the data processing apparatus 100 comprises a decoder 101d (which likewise may be implemented by the processing circuitry 101 of the data processing apparatus 100) configured to generate the SQL query 207 based on the plurality of SQL clauses associated with the plurality of selected SSP labels.

[0037] In an embodiment, the decoder 101 d comprises a set of deterministic rules for generating the SQL query 207 based on the plurality of SQL clauses associated with the plurality of selected SSP labels. In an embodiment, the encoder 101c comprises an encoder machine learning, ML, model, in particular an encoder neural network, configured to select, based on the natural language query 205 and the database schema 206, for each of the plurality of natural language query tokens and each of the plurality of database schema elements an SSP label of the plurality of SSP labels.

[0038] As illustrated in figure 2b, in an embodiment the encoder ML model implemented by the data processing apparatus 100 may be based on a pre-training and fine-tuning approach, where in a first pre-training phase an ML model 220 is pre-trained on a number of basic tasks using a large pretraining dataset, and, subsequently, in a second fine-tuning phase the pre-trained ML model 220 is fine-tuned on a downstream application with the appropriate dataset.

[0039] In the following the general flow of processing the natural language query 205 based on the database schema 206 into the SQL query 207 is described in the context of figure 2b. As already described above, the user 110 may input the natural language query 205 in the user device 111a,b which forwards the natural language query 205 along with the database schema 206 to the data processing apparatus 100. In a further embodiment, the data processing apparatus 100 may be configured to obtain the database schema 206 in a different manner, for instance, directly from the database 203. As already described above, both the natural language query 205 and the database schema 206 are used by the encoder 101c to select for each of the tokens of the natural language query 205 and each of the plurality of database schema elements an SSP label of a plurality of SSP labels, wherein each SSP label is associated with one or more SQL clauses. As illustrated in figure 2b, the encoder 101c may comprise a module 101c-1 for pre-processing the natural language query 205 and a module 101c-2 for pre-processing the database schema information 206. In an embodiment, the pre- processed natural language query 205 and the pre-processed database schema information 206 may be concatenated and encoded by the encoder 101c, i.e. the encoder 101c determines a representation for the combined input 205 and 206. The encoder model representations are used by the SSP predictor 101c-4 of the encoder 101c to output SSPs. The SSPs are fed to the SQL generator 101 d-1 implemented by the decoder, which outputs an SQL query without values (where values are represented with “value” placeholders, e.g. “SELECT * FROM table_1 WHERE age = value”). The correct values corresponding to these placeholders may be provided by the SSP value predictor 101c-3, which in the embodiment shown in figure 2b is implemented by the encoder 101c, to generate the final SQL query 207 that may be returned to the user 110 and / or the database 203 for executing the SQL query 207.

[0040] In an embodiment, the decoder 101d is implemented as a non-autoregressive decoder, which, as will be appreciated, is not a machine learning model. Instead, as already described above, the non-autoregressive decoder 101 d relies on a number of heuristics made on top of the SSP predictions generated by the SSP predictor 101c-4 of the encoder 101c and deterministically and quickly generates the SQL query 207 based on these predictions. Figure 3c illustrates the function of the non-autoregressive decoder 101d implemented by the data processing apparatus 100 according to an embodiment in comparison with other conventional decoding schemes illustrated in figures 3a and 3b.

[0041] As already described above and as will be described in more detail in the context of the example shown in figure 4, the SQL generator 101 d-1 implemented by the data processing apparatus 100 according to an embodiment relies on the presence of a prediction layer provided by the encoder 101c, which predicts SSP labels. As used herein, SSP labels are labels which may be attached to column names and may denote the SQL clauses in which each column should appear. Further details concerning SSP labels, as used herein, may be found in Yu et al., 2021. “GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing”, Proceedings of the 2021 International Conference on Learning Representations (https: / / openreview.net / pdf?id=kyaleYj4zZ), which is fully incorporated herein by reference. In an embodiment, the query encoder 101c-1 may be implemented as a recurrent neural network with or without attention, a self-attentive encoder, an encoder based on pre-trained language models, and / or an encoder leveraging any semantic relations between the input query 205 and the database schema information 206. In an embodiment, the SSP value prediction module 101c-3 of the encoder 101c, which as already described above is configured to replace mentions of the “value” placeholder in the queries generated by the SQL generator 101d-1 , may be implemented as a tagging module operating on top of the query encoder 101c- 1 (as illustrated in the embodiment shown in figure 4), or as an embedding similarity component computing the similarity of candidate n-grams from the input query 205 against the input database’s content or combinations thereof.

[0042] In an embodiment, as already described above, the pre-training dataset generator 101a of the data processing apparatus 100 is configured to make a hypothesis about the existence of a dataset of SQL queries aligned with questions in a source language. A system for identifying and extracting template-pairs of natural language questions with SQL queries can be used on this aligned dataset. An annotator, fluent in both the source and target languages can translate the extracted template-pairs of natural language questions with SQL queries. Finally, a collection of tabular databases can be leveraged for generating natural language question- SQL pairs for the translated templates.

[0043] As will be appreciated, in an embodiment, the SQL generator 101 d- 1 may result in an improved latency. The pre-training dataset generator 101a may contribute to a fast, cost-effective, and low-effort porting of models to new languages. Moreover, the pre-training scheme enables models to operate in a cross-lingual scenario, where the user queries 205 and the database schema information 206 are in different languages.

[0044] In an embodiment, the architecture illustrated in the figure 2b may be implemented using one or more neural networks. For instance, in an embodiment, the architecture illustrated in the figure 2b (i) may be implemented by a pre-trained model based on self-attentive encoder layers (as disclosed in Vaswani, A. et al., 2017. Attention Is All You Need. In: I. Guyon, et al. eds. Advances in Neural Information Processing Systems 30. s.l.: Curran Associates, Inc., pp. 5998-6008) as a backbone for the pre-trainer, (ii) may use the output vectors of this model as vectors to implement the language prediction and Translation Language Modelling tasks, (iii) may use the pre-trained model as a backbone for the encoder 101c, and / or (iv) may implement the SSP prediction module 101c-4 as a feedforward neural network with output dimensionality tailored to the output range. Figure 4 illustrates processing of the query “how many department heads are over 40?” as an exemplary natural language query 205 in the English language. As illustrated in figure 4, the processing circuitry 101 of the data processing apparatus 100 is configured to concatenate the exemplary natural language query 205 “how many department heads are over 40?” with the database schema 206 (also referred to as database schema information 206). More specifically, in the embodiment shown in figure 4, the processing circuitry 101 of the data processing apparatus 100 is configured to place table names, such as “head” between special “<t>” and < / t>” markers. Table names, such as “head”, are followed by the names of all of their columns, like “name”, “age”, “depjd”, and others. All tables include “*” as a column, in order to enable queries of the form “SELECT * from table”.

[0045] As illustrated in figure 4, the such processed concatenated sequence of the exemplary natural language query 205 and the database schema 206 is provided as input to the encoder 101c. As already described above, the encoder 101c of the data processing apparatus 100 is configured to output SSP predictions, i.e. select SSP classification labels (in particular by the module S3.4 illustrated in figure 2b). As an example, according to the exemplary predictions in figure 4, the “WHERE >” SQL clause of the generated SQL query 207 is relevant to the “age” column of the database schema 206. The SQL generator 101 d-1 implemented by the decoder 101 d of the data processing apparatus 100 takes as input the SSP predictions for all tables and SQL clauses and generates the final SQL query 207, after any potential “value” placeholders are replaced by the value mentions predicted by the SSP value predictor 101c- 3.

[0046] In the example shown in figure 4, the SSP predictions may be as follows: o head: FROM o *: SELECT COUNT o age: WHERE >

[0047] The SSP value mention predictions on top of the input query’s, 205, tokens may be as follows: o 40: val_1 , where “val_1” refers to a placeholder for constant values.

[0048] Based on these SSP labels, the SQL generator 101 d-1 implemented by the decoder 101d of the data processing apparatus 100 generates the final SQL query “SELECT COUNT(*) FROM head WHERE age > 40” 207. As will be appreciated, in doing so, the SQL generator 101 d-1 implemented by the decoder 101 d of the data processing apparatus 100 may make use of the fact that the SSP predictions include all the information needed to deterministically construct the SQL query 207. Conventionally, Text-to-SQL ML models are usually trained and tested with data in a single natural language. This allows for easy modelling and experimentation, since language particularities and language-specific semantic representations are not considered and only transduction issues are taken into consideration. However, in a number of implementation scenarios, there is a need to deploy similar models for more than one natural language. While architectures may be reused, in order to adapt a ML model to a new natural language, training data in the new language is required. As will be appreciated, however, successful deep learning models need a large amount of training data, which can be cumbersome to find in multiple natural languages.

[0049] Figure 5 schematically illustrates different processing stages implemented by the data processing apparatus 100 according to an embodiment for generating, based on a training data set in a first natural language, by way of example in English, a training data set in a second natural language, by way of example in Chinese. As illustrated in figure 5, the initial training set comprises, by way of example, a data tuple including the natural language query “How many department heads have age over 40?” 205 and the corresponding SQL query 207 “SELECT COUNT(*) FROM head WHERE age > 40”.

[0050] In a first stage 501 of figure 5, starting from the training set in the original language, e.g. English, the data processing apparatus 100 is configured to obtain a dataset of tuples of the form (natural language query template, SQL query template). For the example shown in figure 5, the corresponding tuple is (“COUNT(*) TABLED have COLO QP0 VALUED”, “SELECT COUNT(*) FROM TABLED WHERE COLO OPO VALUED”), where “head” has been replaced by the data processing apparatus 100 with “TABLED”, “age” with “COLO”, “>” with “OPO”, and “40” with “VALUED”. In an embodiment, the processing circuitry 101 of the data processing apparatus 100 may be configured to extract the (natural language query template, SQL query template) tuples from the original training set based on the template extraction scheme disclosed in Yu, P., Fei, H. & Li, P., 2021. Cross-Lingual Language Model Pretraining for Retrieval. New York, NY, USA, Association for Computing Machinery, p. 1029-1039, which is fully incorporated herein by reference.

[0051] In a second stage 503 of figure 5, translations of the extracted natural language query templates in the first language into the second natural language, e.g. Chinese, are obtained. In an embodiment, the data processing apparatus 100 may be configured to use a machine translation or a human translation to obtain the translations of the extracted natural language query templates into the second natural language, e.g. Chinese. As a result of this second processing stage 503 the dataset consists of a plurality of tuples in the form (natural language query template in the second natural language, SQL query template). For the example shown in figure 5, the corresponding tuple would be (“VALLIEO COLO OPO TABLEO?”,

[0052] “SELECT COUNT(*) FROM TABLEO WHERE COLO OPO VALUEO”).

[0053] In a third stage 505 of figure 5, one or more relational databases in the second natural language, e.g. Chinese, are sampled for table names, column names of tables and / or database elements of tables and multiple instances of each natural language query template in the second natural language are populated based on this sampled data in the second natural language, e.g. Chinese. For the example shown in figure 5, one of the possible tuples obtained in this way would be (“40 W UifW ? ”, “SELECT COUNT(*) FROM head WHERE age > 40”).

[0054] As will be appreciated and as indicated in figure 5, the processing steps 501 , 503, 505 implemented by the data processing apparatus 100 according to an embodiment may be regarded as a kind of distant supervision: a weakly labelled dataset is created in the first natural language (after step 501); a weakly labelled dataset is created in the second natural language (after step 503); and the final dataset is created in the second natural language (after step 505). As a result, this approach that is implemented by the data processing apparatus 100 according to an embodiment requires much less resources than directly translating the entire dataset into the second natural language, because the number of query templates is substantially lower than the number of datapoints, which could be up to several thousands.

[0055] In an embodiment, the encoder ML model of the data processing apparatus 100 is configured to implement cross-lingual capabilities. As illustrated in figure 2b, to this end, the processing circuitry 101 of the data processing apparatus 100 may be configured to implement the pretrainer 101b in a multitask setup including one or more of the following: an SSP pretraining module 101 b-1 , a Translation Language Modelling, TLM, pretraining module 101 b-2 and a Language Prediction, LP, pretraining module 101 b-3.

[0056] In an embodiment, the SSP pretraining module 101 b-1 is configured to pretrain the encoder ML model to correctly predict the SSP labels for each natural language query 205 and database schema 206. Thus, the SSP pretraining module 101 b-1 forces the encoder ML model to learn relations between natural language input queries 205 and database schemata 206. In an embodiment, the Language Prediction, LP, pretraining module 101b-3 is configured to pretrain the encoder ML model to correctly identify the natural language of the natural language query 205. Pretraining this task allows creating a multilingual semantic space and forcing the encoder ML model to learn similar representations for natural language input queries 205 sharing semantics across the involved natural languages. In an embodiment, the gradient after the LP network may be reversed. Further details of such a possible setup of the LP pretraining module 101b-3 may be taken from Sherborne, T. & Lapata, M., 2022. “Zero-Shot Cross-lingual Semantic Parsing”. Dublin, Association for Computational Linguistics, p. 4134-4153, which is herein fully incorporated by reference.

[0057] In an embodiment, the Translation Language Modelling, TLM, pretraining module 101 b-2 is configured to pretrain the encoder ML model to uncover masked tokens in a bilingual input. The intuition behind the implementation of the TLM pretraining module 101 b-2 is to assist the encoder ML model to learn phrase-level or token-level alignment between the inputs in the two languages. Further details about a possible implementation of the TLM, pretraining module 101 b-2 can be found in Conneau and Lample, 2019. In Advances in Neural Information Processing Systems 32. Curran Associates, Inc. p. 7059-7069, which is fully incorporated by reference herein.

[0058] Thus, in an embodiment, the encoder ML model implemented by the encoder 101c of the data processing apparatus 100 is configured to be trained with a composite loss function £, which is a linear combination of the following loss function components: =aSSP - SSP +aLP-^LP +aTLM-^TLM > where £SSPdenotes the loss function component associated with the SSP, £LPdenotes the loss function component associated with the Language Prediction, and £TLMdenotes the loss function component associated with Translation Language Modelling. In this embodiment, the three loss function components are weighted using the weights aSSP, aLP, aTLM, which may be treated as hyperparameters.

[0059] As already described above, in an embodiment the data processing apparatus 100 may be configured to take advantage of transfer learning benefits in that a model 220 may be pretrained on three different tasks: (i) SQL Semantic Prediction, (ii) language prediction and (iii) translation language modelling. More specifically, in an embodiment, the data processing apparatus 100 may make use of a compound objective consisting of three separate crossentropy losses, one for each task. Fine-tuning can be performed solely on the task of SSP, including value mention prediction, or include combinations of the latter two tasks along with SSP.

[0060] According to the embodiments disclosed herein the data processing apparatus 100 is implemented as a pre-training dataset generator, which may generate large-scale datasets in different languages using distant supervision. Such datasets may be used by the data processing apparatus 100 to pre-train models that will, subsequently, be fine-tuned for semantic parsing. As already described above, embodiments of the data processing apparatus 100 implement a multi-task pre-training scenario, which may include two or more cross-lingual tasks that enable a Text-to-SQL model to learn a tighter interaction between the involved languages.

[0061] Figure 6 is a flow diagram illustrating a computer-implemented data processing method 600 for semantic parsing of a natural language query 205 having a plurality of natural language query tokens for generating an SQL query 207 for a relational database 203 having a database schema 206 with a plurality of database schema elements. The computer-implemented data processing method 600 comprises a step 601 of selecting, based on the natural language query 205 and the database schema 206, for each of the plurality of natural language query tokens and each of the plurality of database schema elements an SQL Semantic Predictions, SSP, label of a plurality of SSP labels, wherein each SSP label is associated with at least one SQL clause. Moreover, the computer-implemented data processing method 600 comprises a step 603 of generating the SQL query 207 based on the plurality of SQL clauses associated with the plurality of selected SSP labels. As already described above in the context of the data processing apparatus 100, in an embodiment, the step 603 may include or be followed by a step of replacing the value placeholders in the resulting query.

[0062] As the computer-implemented data processing method 600 can be implemented by the data processing apparatus 100, further features of the computer-implemented data processing method 600 result directly from the functionality of the data processing apparatus 100 and its different embodiments described above and below.

[0063] Embodiments of the data processing apparatus 100 disclosed herein enable very fast and simple decoding, which can enable semantic parsers to serve more requests and more users with less computational requirements than conventional approaches. Moreover, embodiments of the data processing apparatus 100 disclosed herein allow generating training datasets to support new languages with minimal effort. Embodiments of the data processing apparatus 100 disclosed herein further help overcoming the language barrier between database schema and question language.

[0064] The person skilled in the art will understand that the "blocks" ("units") of the various figures (method and apparatus) represent or describe functionalities of embodiments of the present disclosure (rather than necessarily individual "units" in hardware or software) and thus describe equally functions or features of apparatus embodiments as well as method embodiments (unit = step).

[0065] In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus, and method may be implemented in other manners. The described embodiment of an apparatus is merely exemplary. For example, the unit division is merely logical function division and may be another division in an actual implementation. For example, a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may be implemented by using some interfaces. The indirect couplings or communication connections between the apparatuses or units may be implemented in electronic, mechanical, or other forms.

[0066] The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

[0067] In addition, functional units in the embodiments disclosed herein may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units are integrated into one unit.

Claims

CLAIMS1. A data processing apparatus (100) for semantic parsing of a natural language query (205) having a plurality of natural language query tokens for generating a structured query language, SQL, query (207) for a relational database (203) having a database schema (206) with a plurality of database schema elements, wherein the data processing apparatus (100) comprises: an encoder (101c) configured to select, based on the natural language query (205) and the database schema (206), for each of the plurality of natural language query tokens and each of the plurality of database schema elements an SQL Semantic Prediction, SSP, label of a plurality of SSP labels, wherein each SSP label is associated with at least one SQL clause; and a decoder (101 d) configured to generate the SQL query (207) based on the plurality of SQL clauses associated with the plurality of selected SSP labels.

2. The data processing apparatus (100) of claim 1 , wherein the decoder (101 d) comprises a set of deterministic rules for generating the SQL query (207) based on the plurality of SQL clauses associated with the plurality of selected SSP labels.

3. The data processing apparatus (100) of any one of the preceding claims, wherein the encoder (101c) comprises an encoder machine learning, ML, model configured to select, based on the natural language query (205) and the database schema (206), for each of the plurality of natural language query tokens and each of the plurality of database schema elements an SSP label of the plurality of SSP labels.

4. The data processing apparatus (100) of claim 3, wherein the data processing apparatus (100) is further configured to pre-train the encoder ML model based on a loss function, wherein the loss function comprises a loss function component defining a quality measure of the plurality of selected SSP labels based on the natural language query (205) and the database schema (206) of the relational database (203).

5. The data processing apparatus (100) of claim 4, wherein the data processing apparatus (100) is further configured to determine based on the natural language query (205) the natural language of the natural language query (205) and wherein the loss function comprises a further loss function component defining a quality measure of the prediction of the natural language predicted based on the natural language query (205).

6. The data processing apparatus (100) of claim 4 or 5, wherein the natural language query (205) is in a first natural language and in a second natural language and wherein the loss function comprises a further loss function component defining a quality measure of the encoder ML model to uncover masked tokens of the natural language query (205) in the first and the second natural language.

7. The data processing apparatus (100) of any one of claims 3 to 6, wherein the data processing apparatus (100) is further configured to train the encoder ML model based on a training set.

8. The data processing apparatus (100) of claim 7, wherein the data processing apparatus (100) is further configured to: extract from an initial training set in a first natural language a plurality of natural language query templates and a plurality of corresponding SQL query templates; obtain for each of the plurality of natural language query templates in the first natural language a translation into a second natural language for obtaining a plurality of natural language query templates in the second natural language; and generate the training set in the second natural language based on the plurality of natural language query templates in the second natural language and the plurality of corresponding SQL query templates.

9. The data processing apparatus (100) of claim 8, wherein for generating the training set in the second natural language based on the plurality of natural language query templates in the second natural language and the plurality of corresponding SQL query templates the data processing apparatus (100) is configured to sample at least one database in the second natural language for obtaining a plurality of table names, column names of tables and / or database elements of tables from the at least one database in the second natural language and to insert the plurality of table names, column names of tables and / or database elements of tables into the plurality of natural language query templates in the second natural language and the plurality of SQL query templates.

10. The data processing apparatus (100) of claim 7, wherein the data processing apparatus (100) is further configured to: extract from an initial training set a plurality of natural language query templates and a plurality of corresponding SQL query templates; and generate the training set based on the plurality of natural language query templates and the plurality of corresponding SQL query templates by sampling at least one database for obtaininga plurality of table names, column names of tables and / or database elements of tables from the at least one database and to insert the plurality of table names, column names of tables and / or database elements of tables into the plurality of natural language query templates and the plurality of SQL query templates.

11. A data processing method (600) for semantic parsing of a natural language query (205) having a plurality of natural language query tokens for generating a structured query language, SQL, query (207) for a relational database (203) having a database schema (206) with a plurality of database schema elements, wherein the data processing method (600) comprises: selecting (601), based on the natural language query (205) and the database schema (206), for each of the plurality of natural language query tokens and each of the plurality of database schema elements an SQL Semantic Predictions, SSP, label of a plurality of SSP labels, wherein each SSP label is associated with at least one SQL clause; and generating (603) the SQL query (207) based on the plurality of SQL clauses associated with the plurality of selected SSP labels.

12. A computer program product comprising a computer-readable storage medium for storing program code which causes a computer or a processor to perform the method (600) of claim 11 , when the program code is executed by the computer or the processor.

Images