System and method for translating natural language sentences into database queries

Abstract

The described system and method allow automatic translation from a natural language (e.g., English) into an artificial language such as a structured query language SQL. In some embodiments, a translator module includes an encoder component and a decoder component, both of which include a recursive neural network. Training the translator module includes two stages. The first stage trains the translator module to produce an AL output when presented with an artificial language AL input. For example, the translator is first trained to regenerate the AL input. The second stage of training includes training the translator to produce an AL output when presented with a natural language NL input.

Description

Technical Field

[0001] The present invention relates to systems and methods for automatic translation from natural language to artificial machine-readable language. Background Art

[0002] In recent years, an increasing number of products and services rely on collecting and analyzing large amounts of data. Examples cover nearly all areas of human activity, from manufacturing to commerce, scientific research, healthcare, and defense. They include, for example: retail systems that manage inventory, customers, and sales across multiple stores and warehouses; logistics software that manages large and diverse fleets of carriers; and Internet advertising services that rely on user profiles to target offers to potential customers. Managing large amounts of data has promoted innovation and development in database architectures and systems and methods for interacting with the corresponding data. As databases increase in size and complexity, it becomes impractical to use human operators to quickly search, retrieve, and analyze data.

[0003] At the same time, we are witnessing an explosive growth and diversification of electronic appliances, commonly referred to as the "Internet of Things". Devices ranging from mobile phones to home appliances, wearable devices, entertainment devices, and various sensors and gadgets incorporated into cars, houses, etc. are often connected to remote computers and / or various databases to perform their functions. A highly desirable feature of such devices and services is user-friendliness. Commercial pressure to make such products and services accessible to the general public is driving research and development of innovative human-machine interfaces. Some examples of such topologies include personal assistants, such as Apple's and of and others.

[0004] There is therefore considerable interest in developing systems and methods that facilitate interaction between humans and computers, particularly in applications involving database access and / or management. Summary of the invention

[0005] According to one aspect, a method includes employing at least one hardware processor of a computer system to: execute an artificial language (AL) encoder and a decoder coupled to the AL encoder, the AL encoder configured to receive a first input array including a representation of an input AL sentence expressed in an artificial language, and in response generate a first internal array. The decoder is configured to receive the first internal array and in response generate a first output array including a representation of a first output AL sentence expressed in the artificial language. The method further includes: determining a first similarity score indicating a similarity between the input AL sentence and the first output AL sentence in response to providing the first input array to the AL encoder; and adjusting a first set of parameters of the decoder according to the first similarity score to improve a match between an AL encoder input and a decoder output. The method further includes: determining whether a first stage training termination condition is satisfied; and in response, executing an NL encoder configured to receive a second input array including a representation of an input natural language (NL) sentence expressed in a natural language, and in response outputting a second internal array to the decoder if the first stage training termination condition is satisfied. The method further includes: determining a second output array generated by the decoder in response to receiving the second internal array, the second output array including a representation of a second output AL sentence expressed in the artificial language; and determining a second similarity score indicating a similarity between the second output AL sentence and a target AL sentence, the target AL sentence including a translation of the input NL sentence into the artificial language. The method further includes: adjusting a second set of parameters of the NL encoder based on the second similarity score to improve the match between the decoder output and the target output, the target output representing a corresponding translation of the input received by the NL encoder into the artificial language.

[0006] According to another aspect, a computer system includes at least one hardware processor and memory, the at least one hardware processor configured to: execute an AL encoder and a decoder coupled to the AL encoder, the AL encoder configured to receive a first input array including a representation of an input AL sentence expressed in an artificial language, and in response generate a first internal array. The decoder is configured to receive the first internal array and in response generate a first output array including a representation of a first output AL sentence expressed in the artificial language. The at least one hardware processor is further configured to: determine a first similarity score indicating a similarity between the input AL sentence and the first output AL sentence in response to providing the first input array to the AL encoder; and adjust a first set of parameters of the decoder according to the first similarity score to improve the match between the AL encoder input and the decoder output. The at least one hardware processor is further configured to: determine whether a first stage training termination condition is satisfied; and in response, execute an NL encoder configured to receive a second input array including a representation of an input NL sentence expressed in a natural language, and in response output a second internal array to the decoder if the first stage training termination condition is satisfied. The at least one hardware processor is further configured to: determine a second output array generated by the decoder in response to receiving the second internal array, the second output array comprising a representation of a second output AL sentence expressed in the artificial language. The at least one hardware processor is further configured to: determine a second similarity score indicating a similarity between the second output AL sentence and a target AL sentence, the target AL sentence comprising a translation of the input NL sentence into the artificial language; and adjust a second set of parameters of the NL encoder based on the second similarity score to improve a match between a decoder output and a target output representing a corresponding translation of an input received by the NL encoder into the artificial language.

[0007] According to another aspect, a non-transitory computer-readable medium stores instructions that, when executed by a first hardware processor of a first computer system, cause the first computer system to form a trained translator module comprising an NL encoder and a decoder connected to the NL encoder, and wherein training the translator module comprises employing a second hardware processor of a second computer system to: couple the decoder to an AL encoder, the AL encoder configured to receive a first input array comprising a representation of an input AL sentence expressed in an artificial language, and in response generate a first internal array. The AL encoder is coupled to the decoder such that the decoder receives the first internal array and in response generates a first output array comprising a representation of a first output AL sentence expressed in the artificial language. Training the translator module further comprises: determining a first similarity score indicating a similarity between the input AL sentence and the first output AL sentence in response to providing the first input array to the AL encoder; and adjusting a first set of parameters of the decoder according to the first similarity score to improve matching between AL encoder input and decoder output. Training the translator module further includes: determining whether a first stage training termination condition is satisfied; and in response, coupling the NL encoder to the decoder if the first stage training termination condition is satisfied, so that the NL encoder receives a second input array including a representation of an input NL sentence expressed in a natural language, and in response, outputs a second internal array to the decoder. Training the translator module further includes: determining a second output array generated by the decoder in response to receiving the second internal array, the second output array including a representation of a second output AL sentence expressed in the artificial language; and determining a second similarity score indicating a similarity between the second output AL sentence and a target AL sentence, the target AL sentence including a translation of the input NL sentence into the artificial language. Training the translator module further includes: adjusting a second set of parameters of the NL encoder based on the second similarity score to improve the match between the decoder output and the target output, the target output representing a corresponding translation of the input received by the NL encoder into the artificial language.

[0008] According to another aspect, a computer system includes a first hardware processor configured to execute a trained translator module including an NL encoder and a decoder connected to the NL encoder, wherein training the translator module includes employing a second hardware processor of a second computer system to: couple the decoder to an AL encoder, the AL encoder configured to receive a first input array including a representation of an input AL sentence expressed in an artificial language, and in response generate a first internal array. The AL encoder is coupled to the decoder such that the decoder receives the first internal array and in response generates a first output array including a representation of a first output AL sentence expressed in the artificial language. Training the translator module further includes: determining a first similarity score indicating a similarity between the input AL sentence and the first output AL sentence in response to providing the first input array to the AL encoder; and adjusting a first set of parameters of the decoder based on the first similarity score to improve matching between AL encoder input and decoder output. Training the translator module further includes: determining whether a first stage training termination condition is satisfied; and in response, coupling the NL encoder to the decoder if the first stage training termination condition is satisfied, so that the NL encoder receives a second input array including a representation of an input NL sentence expressed in a natural language, and in response, outputs a second internal array to the decoder. Training the translator module further includes: determining a second output array generated by the decoder in response to receiving the second internal array, the second output array including a representation of a second output AL sentence expressed in the artificial language. Training the translator module further includes: determining a second similarity score indicating a similarity between the second output AL sentence and a target AL sentence, the target AL sentence including a translation of the input NL sentence into the artificial language; and adjusting a second set of parameters of the NL encoder based on the second similarity score to improve the match between the decoder output and the target output, the target output representing a corresponding translation of the input received by the NL encoder into the artificial language. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The foregoing aspects and advantages of the present invention will become better understood after reading the following detailed description and after referring to the accompanying drawings, in which:

[0010] Figure 1 An exemplary automated database access system is shown in which a set of clients cooperates with a translator training system and a database server according to some embodiments of the invention.

[0011] Figure 2-A An exemplary hardware configuration of a client system according to some embodiments of the present invention is shown.

[0012] Figure 2-B An exemplary hardware configuration of a translator training system according to some embodiments of the present invention is shown.

[0013] Figure 3 An exemplary set of software components executing on a client system according to some embodiments of the invention is presented.

[0014] Figure 4 An exemplary data exchange between a client system and a database server according to some embodiments of the present invention is shown.

[0015] Figure 5 Illustrated are exemplary components of a translator training system according to some embodiments of the invention.

[0016] Figure 6 An exemplary translator training procedure according to some embodiments of the invention is shown.

[0017] Figure 7 An exemplary operation of a translator module according to some embodiments of the invention is illustrated.

[0018] Figure 8 Illustrated are exemplary components and operations of a translator module according to some embodiments of the invention.

[0019] Fig. 9 An exemplary sequence of steps performed by a translator training system according to some embodiments of the invention is shown.

[0020] Fig.10 An exemplary first stage of training a translator module according to some embodiments of the invention is illustrated.

[0021] Figure 11-A An exemplary second stage of training a translator module according to some embodiments of the invention is illustrated.

[0022] Figure 11-B An alternative exemplary second stage of training a translator module according to some embodiments of the invention is illustrated.

[0023] Fig.12 An exemplary sequence of steps for training a translator module based on a plurality of training corpora according to some embodiments of the invention is shown. DETAILED DESCRIPTION

[0024] In the following description, it should be understood that all referenced connections between structures can be direct operational connections or indirect operational connections through intermediate structures. A set of elements includes one or more elements. Any reference to an element should be understood to refer to at least one element. A plurality of elements include at least two elements. Unless otherwise required, any described method steps do not necessarily need to be performed in a specific illustrated order. A first element (e.g., data) derived from a second element includes a first element equal to the second element and a first element and optionally other data generated by processing the second element. Making a determination or decision based on a parameter includes making a determination or decision based on the parameter and optionally based on other data. Unless otherwise specified, an indicator of some quantity / data may be the quantity / data itself, or an indicator different from the quantity / data itself. A computer program is a sequence of processor instructions that performs a task. The computer program described in some embodiments of the present invention may be a stand-alone software entity or sub-entity (e.g., subroutine, library) of other computer programs. The term "database" is used herein to represent any organized data set. Unless otherwise specified, a sentence is a sequence of words and / or symbols expressed in a natural or artificial language. Two sentences expressed in different languages ​​are considered herein to be translations of each other when the two sentences are semantically equivalent forms of each other (i.e., the two sentences have the same or very similar meaning). Computer-readable media encompass non-transitory media such as magnetic, optical, and semiconductor storage media (e.g., hard drives, optical disks, flash memories, DRAMs) and communication links such as conductive cables and optical fiber links. According to some embodiments, the present invention provides, among other things, a computer system comprising hardware (e.g., one or more processors) programmed to perform the methods described herein, and a computer-readable medium encoding instructions for performing the methods described herein.

[0025] The following description illustrates embodiments of the invention by way of example and not necessarily by way of limitation.

[0026] Figure 1 An exemplary database access and management system according to some embodiments of the present invention is shown. A plurality of client systems 12a to d may interact with a database server 18, such as executing queries thereby accessing / retrieving / writing a set of data from / to a database 20. Exemplary databases 20 include relational databases, extensible markup language (XML) databases, spreadsheets, and key-value stores, among others.

[0027] Exemplary client systems 12a-d include personal computer systems, mobile computing platforms (laptops, tablets, mobile phones), entertainment devices (TVs, game consoles), wearable devices (smart watches, fitness bands), home appliances, and any other electronic device that includes a processor, memory, and a communication interface. Client systems 12a-d are connected to a server 18 via a communication network 14 (e.g., the Internet). Portions of network 14 include a local area network (LAN), such as a home or corporate network. Database server 18 generally describes a set of computing systems that are communicatively coupled to a database 20 and are configured to access the database 20 to perform data insertion, data retrieval, and / or other database management operations.

[0028] In one exemplary application of the illustrated system, client systems 12a-d represent individual computers used by employees of an e-commerce company, and database 20 represents a relational database storing records of products being sold by the corresponding company. An employee may use the illustrated system, for example, to find out how many items of a particular product are currently in stock at a particular warehouse.

[0029] In some embodiments, access to database 20 is facilitated by software executed on client systems 12a-d and / or database server 18, the software including software that enables sentences expressed in a natural language (e.g., English, Chinese) to be automatically translated into a language such as Structured Query Language (SQL), a programming language (e.g., C++, In some embodiments, the translator comprises a set of sentences expressed in an artificial, formal language such as XML (bytecode) and / or a markup language (e.g., XML, Hypertext Markup Language - HTML). In some embodiments, the corresponding translator comprises an artificial intelligence system, such as a set of networks trained by a translator training system 16 that is also connected to the network 14. The operation of the translator training system 16 and the translator itself will be described in more detail below.

[0030] Figure 2-A An exemplary hardware configuration of the client system 12 is shown. The client system 12 may represent Figure 1 Any of the client systems 12a to d. Without loss of generality, the illustrated client system is a computer system. The hardware configuration of other client systems (e.g., mobile phones, smart watches) may be slightly different. Figure 2-A. The client system 12 includes a set of physical devices, including a hardware processor 22 and a memory unit 24. The processor 22 includes a physical device (e.g., a microprocessor, a multi-core integrated circuit formed on a semiconductor substrate, etc.) configured to perform computational and / or logical operations with the aid of a set of signals and / or data. In some embodiments, such operations are delivered to the processor 22 in the form of a sequence of processor instructions (e.g., machine code or other types of encoding). The memory unit 24 may include a volatile computer-readable medium (e.g., DRAM, SRAM) that stores instructions and / or data accessed or generated by the processor 22.

[0031] Input devices 26 may include computer keyboards, mice, and microphones, among others, including corresponding hardware interfaces and / or adapters that allow a user to introduce data and / or instructions into client system 12. Output devices 28 may include display devices (such as monitors and speakers, among others) and hardware interfaces / adapters (such as graphics cards) that allow client system 12 to pass data to the user. In some embodiments, input devices 26 and output devices 28 may share a common piece of hardware, such as in the case of a touch screen device. Storage devices 32 include computer-readable media that achieve non-volatile storage, reading, and writing of software instructions and / or data. Exemplary storage devices 32 include magnetic and optical disks and flash memory devices and removable media (such as CD and / or DVD disks and drives). The set of network adapters 34 enables client system 12 to connect to a computer network and / or other devices / computer systems. Controller hub 30 represents the multiple system, peripheral, and / or chipset buses and / or all other circuit systems that achieve communication between processor 22 and devices 24, 26, 28, 32, and 34. For example, controller hub 30 may include a memory controller, an input / output (I / O) controller, and an interrupt controller, among others. In another example, controller hub 30 may include a north bridge connecting processor 22 to memory 24 and / or a south bridge connecting processor 22 to devices 26, 28, 32, and 34.

[0032] Figure 2-BAn exemplary hardware configuration of a translator training system 16 according to some embodiments of the present invention is shown. The illustrated training system includes a computer that includes at least a training processor 122 (e.g., a microprocessor, a multi-core integrated circuit), a physical memory 124, a set of training storage devices 132, and a set of training network adapters 134. The storage devices 132 include computer-readable media that enable non-volatile storage, reading, and writing of software instructions and / or data. The adapters 134 may include a network card and other communication interface that enables the training system 16 to connect to the communication network 14. In some embodiments, the translator training system 16 further includes input devices and output devices that may be functionally similar to the input devices 26 and output devices 28 of the client system 12, respectively.

[0033] Figure 3 An exemplary computer program executed on client system 12 according to some embodiments of the present invention is shown. This software may include an operating system (OS) 40, which may include any widely available operating system, such as Microsoft or Android TM and others. OS 40 provides an interface between the hardware of client system 12 and a set of applications including, for example, translation application 41. In some embodiments, application 41 is configured to automatically translate natural language (NL) sentences into artificial language (AL) sentences, such as into a set of SQL queries and / or into a sequence of software instructions (code). Translation application 41 includes a translator module 62 that performs the actual translation, and may further include components that receive corresponding natural language sentences from a user (e.g., as text or language via input device 26), components that parse and analyze corresponding NL input (e.g., speech analyzer, word segmenter, various dictionaries, etc.), components that transmit the translated AL output to database server 18, and components that display the content of the response from server 18 to the user, and others. Translator module 62 includes an example of an artificial intelligence system (e.g., a set of neural networks) trained by translator training system 16 to perform NL to AL translation as further described below. This training may produce a set of optimal parameter values ​​for the translator 62, values ​​that may be communicated from the training system 16 to the client 12 and / or database server 18, e.g., via periodic or on-demand software updates. The term "trained translator" herein refers to a translator module instantiated with such optimal parameter values ​​received from the translator training system 16.

[0034] For clarity, the following description will focus on an exemplary application in which the translator module 62 outputs a database query, i.e., a set of sentences expressed in a query language such as SQL. The illustrated systems and methods are thus directed to enabling a human operator to perform database queries. However, the skilled person will appreciate that the described systems and methods may be modified and adapted to other applications in which the translator is configured to generate computer code (e.g., bytecode, etc.), data markup (e.g., XML), or output expressed in any other artificial language.

[0035] Figure 4 An exemplary data exchange between a client system 12 and a database server 18 according to some embodiments of the present invention is shown. The client system 12 sends a query 50 to the database server 18, and in response receives a query result 52 comprising the result of executing the query 50. The query 50 comprises an encoding of a set of instructions that, when executed by the server 18, causes the server 18 to perform certain manipulations of the database 20 (e.g., selectively inserting data into the database 20 or selectively retrieving data from the database 20, respectively). The query result 52 may comprise, for example, an encoding of a set of database records selectively retrieved from the database 20 according to the query 50.

[0036] In some embodiments, the query 50 is expressed in an artificial language such as SQL. In an alternative embodiment, the query 50 may be expressed as a set of natural language sentences. In these embodiments, the translator module as described herein may be executed on the database server 18.

[0037] Figure 5 Exemplary components of a translator training system according to some embodiments of the invention are illustrated. The training system 16 may execute a translator training engine 60, which includes an instance of a translator module 62 and a training module 64 connected to the translator module 62. In some embodiments of the invention, the engine 60 is communicatively coupled to a set of training corpora 17 employed by the training module 64 to train the translator module 62. The corpora 17 may include at least one artificial language (AL) training corpus 66, and / or a set of natural language to artificial language (NL-AL) training corpora 68a-b.

[0038] In some embodiments, the AL training corpus 66 includes multiple entries all expressed in the same artificial language. In one example, each entry consists of at least one AL statement generated automatically or by a human operator. Some entries may contain multiple AL statements, some of which are considered synonyms or semantically equivalent to each other. In an example where the corresponding AL is a database query language, two AL statements may be considered synonyms if they result in the retrieval of the same data from the database. In a similar manner, in a programming language example, two AL statements (i.e., code snippets) may be synonyms / semantically equivalent if they produce the same computational results.

[0039] In some embodiments, NL-AL corpora 68a to b include multiple entries, each of which consists of a tuple (e.g., a pair) of sentences, wherein at least one sentence is expressed in an artificial language, and another sentence is expressed in a natural language. In some embodiments, the AL side of the tuple includes translating the NL side of the corresponding tuple into the artificial language. In other words, the corresponding AL side of the tuple has the same or very similar meaning as the NL side of the corresponding tuple. Some NL-AL tuples may consist of one NL sentence and multiple synonymous AL sentences. Other NL-AL tuples may have multiple NL sentences corresponding to one AL sentence. Different NL-AL corpora 68a to b may correspond to different natural languages ​​(e.g., English to Chinese). In another example, different NL-AL corpora may contain different groups of NL sentences expressed in the same natural language (e.g., English). In one such example, one NL-AL corpus is used to train a translator to be used by an English-speaking sales representative, and another NL-AL corpus can be used to train a translator for use by an English-speaking database manager.

[0040] The training module 64 is configured to train the translator module 62 to produce a desired output, e.g., correctly translate a natural language sentence into an artificial language sentence, as will be seen in more detail below. Training generally refers herein to the process of adjusting a set of parameters of the translator module 62 in an effort to obtain a desired result (e.g., a correct translation). Figure 6302 to 304 can select corpus items (e.g., natural language sentences) and input the corresponding corpus items to the translator module 62. The module 62 can then generate outputs according to the received inputs. Step 308 compares the corresponding outputs with the desired outputs and determines a performance score, such as a translation error indicating the similarity between the actual output of the module 62 and the desired output. In response to determining the performance score, in step 310, the training module 64 can update the parameters of the module 62 in a manner that increases the performance of the translator module 62 (e.g., by reducing the translation error). This parameter adjustment can proceed according to any method known in the art. Some examples include back propagation using gradient descent, simulated annealing, and genetic algorithms. In some embodiments, training ends when a certain termination condition is met (step 312). More details about the termination condition are given below.

[0041] In some embodiments, after training is complete, the translator training system 16 outputs a set of translator parameter values ​​69 in step 314. When the module 62 includes an artificial neural network, the translator parameter values ​​69 may include, for example, a set of synaptic weights and / or a set of network architecture parameter values ​​(e.g., a number of layers, a number of neurons per layer, a contact map, etc.). The parameter values ​​69 may then be transmitted to the client systems 12a-d and / or the database server 18 and used to instantiate a corresponding local translator module that performs automated natural to artificial language translation.

[0042] Figure 7 The diagram illustrates an exemplary operation of the translator module 62 according to some embodiments of the present invention. The module 62 is configured to automatically translate a natural language (NL) sentence, such as the exemplary sentence 54, into an artificial language (AL) sentence, such as the exemplary sentence 56. The term 'sentence' is used herein to refer to any sequence of words / symbols expressed in a natural or artificial language. Examples of natural languages ​​include English, German, and Chinese. Artificial languages ​​include a set of symbols (e.g., keywords, identifiers, operators) and a set of rules for combining the corresponding symbols. The rules are usually called grammars or syntax and are usually language specific. Exemplary artificial languages ​​include formal computer languages, such as query languages ​​(e.g., SQL), programming languages ​​(e.g., C++, Perl, bytecode) and markup languages ​​(e.g., XML, HTML). Exemplary NL sentences include statements, questions, and commands, among others. For example, exemplary AL sentences include a computer code and an SQL query.

[0043] In some embodiments, the translator module 62 receives an input array 55 including a computer-readable representation of an NL sentence, and generates an output array 57 including an encoding of the AL sentence generated by translating the input NL sentence. The input and / or output arrays may include numerical arrays calculated using any method known in the art, such as one-hot encoding. In one such example, each word of the NL vocabulary is assigned a different numeric label. For example, 'how' may have a label 2 and 'many' may have a label 37. Then, the one-hot representation of the word 'how' may include a binary N x 1 vector, where N is the size of the vocabulary, and where all elements are 0, except for the second element (which has a value of 1). At the same time, the word 'many' may be represented as a binary N x 1 vector, where all elements are 0, except for the 37th element. In some embodiments, the input array 55 encoding a sequence of words such as the input sentence 54 includes an N x M binary array, where M represents the count of the words of the input sentence, and where each row of the input array 55 represents a different word of the input sentence. Continuous rows of the input array 55 may correspond to continuous words of the input sentence. Output array 57 may use a similar one-hot encoding strategy, although the vocabularies used to encode the input and output may be different from each other. Since input array 55 and output array 57 represent input sentence 54 and output sentence 56, respectively, output array 57 will be considered herein as a translation of input array 55.

[0044] Transforming the NL sentences 54 into the input array 55 and the output array 57 into the AL sentences 56 may include operations that may be performed by software components separate from the translator module 62, such as parsing, symbolization, etc.

[0045] In some embodiments, module 62 includes an artificial intelligence system, such as an artificial neural network trained to perform the illustrated translation. Such artificial intelligence systems may be constructed using any method known in the art. Figure 8 In the preferred embodiment illustrated in FIG. 6 , module 62 includes an encoder 70 and a decoder 72 coupled to the encoder 70. Each of the encoder 70 and the decoder 72 may include a neural network, such as a recurrent neural network (RNN). RNNs form a special class of artificial neural networks in which the connections between the network nodes form a directed graph. Examples of recurrent neural networks include long short-term memory (LSTM) networks, among others.

[0046] The encoder 70 receives the input array 55 and outputs an internal array 59, which includes an internal representation of the input sentence 54 owned by the translator module. In practice, the internal array 59 includes a mathematical transformation of the input array 55 via a set of operations (e.g., matrix multiplication, application of activation functions, etc.) that are unique to the encoder 70. In some embodiments, the size of the internal array 59 is fixed, while the size of the input array 55 can vary according to the input sentence. For example, a relatively large input array can be used to represent a long input sentence compared to a short input sentence. From this perspective, it can be said that in some embodiments, the encoder 70 transforms a variable size input into a fixed size encoding of the corresponding input. In some embodiments, the decoder 72 regards the internal array 59 as an input and generates an output array 57 through a second set of mathematical operations. The size of the output array 57 can vary according to the content of the internal array 59 and therefore according to the input sentence.

[0047] Fig. 9 The diagram illustrates an exemplary process of training the translator module 62 to perform automated NL to AL translation. In some embodiments, the training includes at least two stages. The first stage represented by steps 322 to 324 includes training the translator module 62 based on an artificial language corpus (e.g., Figure 5 In some embodiments, step 322 includes training the translator module 62 to produce an AL output when fed with an AL input. In one such example, the translator module 62 is trained to regenerate an input expressed in a corresponding artificial language. In another example, the module 62 is trained to produce a synonym / semantic equivalent form of the corresponding input when fed with an AL input. In yet another example, the module 62 is trained so that its output is at least grammatically correct, that is, the output complies with the grammar / syntax rules of the corresponding artificial language.

[0048] In some embodiments, the first phase of training continues until a set of termination conditions are met (step 324). The termination conditions may include performance criteria, such as whether the average deviation from the expected output of module 62 (i.e., the translation error) is less than a predetermined threshold. Another exemplary performance criterion includes whether the output of translation module 62 is mostly syntactically correct, for example, at least 90% of the time. In embodiments where the output of module 62 is expressed in a programming language, testing syntactic correctness may include attempting to compile the corresponding output and determining that the corresponding output is correct when there are no compilation errors. Other exemplary termination conditions include computational cost criteria, for example, training may continue until a predetermined time limit or iteration count has been exceeded.

[0049] In some embodiments, Fig. 9The second phase of training illustrated in steps 326-328 of 16 includes training the translator module 62 based on a natural language to artificial language corpus (i.e., using NL-AL tuples). In one such example, the module 62 is trained to output the corresponding AL side of the tuple when fed the NL side of the tuple as input. In an alternative embodiment, the module 62 may be trained to output at least a synonym for the AL side of the corresponding tuple. The second phase of training may continue until a termination criterion is met (e.g., until a desired percentage of correct NL to AL translations is achieved). Next, in step 330, the translator training system 16 may output translator parameter values ​​69 resulting from the training. In an exemplary embodiment in which the translator module 62 uses a neural network, the parameter values ​​69 may include the values ​​of the synaptic weights obtained via training.

[0050] Fig.10 Further illustrated is the first phase training in a preferred embodiment of the present invention. The illustrated first phase translator module 62a includes an artificial language encoder 70a connected to a decoder 72. The AL encoder 70a obtains an input array 55a and outputs an internal array 59a, which is transformed into an output array 57a by the decoder 72. In some embodiments, the first phase training includes providing a plurality of AL inputs to the translator module 62a, and tuning the parameters of the module 62a to produce an AL output similar to the corresponding presented input. In other words, in some embodiments, the goal of the first phase training can be to make the output more similar to the input. In an alternative embodiment, the training goal can be that the output is at least a synonym of the corresponding input, or the output is grammatically correct in the corresponding artificial language.

[0051] In one example of the first stage of training, for each pair of input / output arrays, the training module 64 may calculate a value indicating the difference between the output array and the input array (in Fig.10 57a and 55a, respectively. The similarity metric can be calculated using any method known in the art, such as based on the Manhattan or edit distance between the input array and the output array. Training module 64 can then adjust the parameters of AL encoder 70a and / or decoder 72 to increase the similarity between the output of decoder 72 and the input of AL encoder 70a, for example, to reduce the average Manhattan distance between arrays 55a and 57a.

[0052] The first phase training may continue until a first phase termination condition is met (eg, until a predetermined performance level is achieved, until all members of the AL corpus 66 have been used in training, etc.).

[0053] Figure 11-AThe diagram illustrates an exemplary second stage training process, which includes training to translate between the natural language and artificial language used in the first stage training. In some embodiments, progressing from the first stage to the second stage includes switching to the second stage translator module 62b obtained by replacing the AL encoder 70a with a natural language encoder 70b while retaining the trained decoder 72. In other words, the decoder 72 keeps exemplifying the parameter values ​​generated by the first stage training. The architecture and / or parameter values ​​of the NL encoder 70b may be substantially different from the architecture and / or parameter values ​​of the AL encoder 70a. One reason for this difference is that the vocabularies of artificial and natural languages ​​are usually different from each other, so that the input array representing the NL sentence may be different from the input array representing the AL sentence at least in terms of size. Another reason why encoders 70a to 70b may have different architectures is that the grammar / syntax of the artificial language is usually substantially different from the grammar / syntax of the natural language.

[0054] NL encoder 70b obtains input array 55b representing NL sentences and outputs internal array 59b. In some embodiments, internal array 59b has the same size and / or structure as internal array 59a output by AL encoder 59a of translation module 62a (see Fig.10 ). The internal array 59b is then fed as input to a decoder 72 which in turn produces an output array 57c representing the AL sentence.

[0055] In some embodiments, the second stage training uses NL-AL tuples, where the AL side of the tuple represents the target AL that is translated into the NL side of the corresponding tuple. The second stage training may include providing multiple NL inputs to the NL encoder 70b, where each NL input includes the NL side of the NL-AL tuple, and tuning the parameters of the translator module 62b so that the output of the decoder 72 is similar to the AL side of the corresponding NL-AL tuple. In other words, the goal of the second stage training is to make the output of the decoder 72 more similar to the translation of the corresponding NL input to the target AL.

[0056] In one exemplary embodiment, in response to the NL side of each tuple (denoted as Figure 11-A 57b) is fed to the NL encoder 70b, the training module 64 may compare the output of the decoder 72 (array 57c) with the AL side of the corresponding tuple (array 57b). The comparison may include calculating a similarity metric indicating the similarity between arrays 57b and 57c. The training module 64 may then adjust the parameters of the NL encoder 70b and / or decoder 72 in a direction that increases the similarity between arrays 57b and 57c.

[0057] exist Figure 11-BAn alternative scenario for the second stage of training is illustrated in FIG. This alternative scenario employs a (trained) AL encoder obtained via the first stage of training (e.g., exemplified with parameter values ​​resulting from the first stage of training). Fig.10 In some embodiments, the NL encoder 70b is fed with an input array 55b representing the NL side of the NL-AL tuple, while the AL encoder is fed with an output array 57b representing the AL side of the corresponding tuple. The method relies on the observation that the AL encoder 70a has been configured during the first stage of training to transform the AL input into a 'proper' internal array 59c, which the decoder 72 can then transform back into the corresponding AL input. In other words, in order for the decoder 72 to produce the output array 57a, its input must be as close as possible to the output of the (already trained) AL encoder 70a. Therefore, in Figure 11-B In the embodiment illustrated in , training module 64 may compare the output of NL encoder 72 (i.e., internal array 59b) with internal array 59c and quantize the difference as a similarity measure. Training module 64 may then adjust parameters of NL encoder 70b and / or decoder 72 in a direction that increases the similarity between arrays 59b and 59c.

[0058] Fig.12 An exemplary sequence of steps for training a translator module 62 from multiple training corpora is shown according to some embodiments of the present invention. The illustrated method relies on the observation that a decoder 72 can be trained only once for each target artificial language (see first stage training above), and then reused in an already trained form to derive multiple translator modules, e.g., modules that can translate from multiple source natural languages ​​(e.g., English, German, etc.) into respective target artificial languages ​​(e.g., SQL).

[0059] In another example, each different transliterator module may be trained on a different set of NL sentences expressed in the same natural language (e.g., English). This particular embodiment relies on the observation that language is often professional and task-specific, i.e., the sentences / commands used by human operators to solve certain problems are different from those used in other situations. Therefore, some embodiments employ one corpus (i.e., a set of NL sentences) to train a transliterator to be used by sales personnel, and another corpus to train a transliterator to be used by technicians.

[0060] Fig.12Steps 342 to 344 in illustrate the first phase training process, including training the AL encoder 70a and / or the decoder 72. In response to a successful first phase training, step 346 replaces the AL encoder 70a with a NL encoder. In some embodiments, a second phase training of the NL encoder is then performed for each available NL-AL corpus. When switching from one NL-AL corpus to another NL-AL corpus (e.g., switching from English to Spanish, or from 'sales English' to 'technical English'), some embodiments replace the existing NL encoder with a new NL encoder suitable for the current NL-AL corpus (step 358), while retaining the already trained decoder 72. Such optimizations can greatly facilitate and accelerate the training of the automatic translator.

[0061] In some embodiments, the second stage training only adjusts the NL encoder 70b (see Figures 11-A to 11-B ) and the parameters of the decoder 72 are fixed at the values ​​obtained via the first phase training. This training strategy aims to maintain the performance of the decoder 72 at the level achieved by the first phase training regardless of the choice of the source natural language or NL-AL corpus. In other embodiments where the training includes adjusting the parameters of both the NL encoder 70b and the decoder 72, step 358 may further include resetting the parameters of the decoder 72 to the values ​​obtained at the end of the first phase training.

[0062] The exemplary systems and methods described above allow for automatic translation from a source natural language, such as English, into a target artificial language (e.g., SQL, a programming language, a markup language, etc.). One exemplary application of some embodiments of the present invention allows a layperson to perform database queries using simple questions expressed in natural language without the need to know a query language such as SQL. For example, a sales operator may ask a client machine "How many customers are under 30 years old in Colorado?" In response, the machine may translate the corresponding question into a database query and execute the corresponding query to retrieve an answer to the operator's question.

[0063] Some embodiments use a translator module to translate the NL sequence of words into an AL sentence, for example, into a valid query that can be used to selectively retrieve data from a database. The translator module may include a set of artificial neural networks, such as an encoder network and a decoder network. The encoder and decoder may be constructed using a recurrent neural network (RNN) or any other artificial intelligence technology.

[0064] In some embodiments, the training translator module includes at least two stages. In the first stage, the translator module is trained to produce AL output in response to AL input. For example, the first stage training may include training the translator module to regenerate AL input. In an alternative embodiment, the translator is trained to produce grammatically correct AL sentences in response to AL input. Using a convenient metaphor, it can be said that the first stage training teaches the translator module to 'speak' the corresponding artificial language. In practice, the first stage training includes presenting a huge corpus (e.g., SQL query) of AL sentences to the translator. For each input sentence, the output of the translator is evaluated to determine a performance score, and the parameters of the translator are adjusted to improve the performance of the translator in terms of training.

[0065] The subsequent second stage includes training the translator module to generate AL output in response to NL input expressed in the source language. The training of the second stage can adopt an NL-AL corpus including multiple sentence tuples (e.g., pairs), each tuple having at least an NL side and an AL side. In an exemplary embodiment, each AL side of the tuple can represent the translation of the corresponding NL side, that is, the desired output of the translator when the corresponding NL side of the tuple is presented. The exemplary second stage training proceeds as follows: For each NL-AL tuple, the translator receives the NL side as input. The output of the translator is compared with the AL side of the tuple to determine the translation error, and the parameters of the translator are adjusted to reduce the translator error.

[0066] Conventionally, several pairs of projects are used to train conventional automatic translators, wherein one member of the pair is expressed in the source language, and the other member of the pair is expressed in the target language. A technical obstacle in the face of this conventional training is the size of the training corpus. It is generally acknowledged in the art that larger and more diverse corpora produce more robust and high-performance translators. Achieving reasonable translation performance may require tens of thousands of NL-AL tuples or more. However, due to the general inability to automatically generate NL-AL tuples, it is unrealistic to set the required technical manpower workload of this large corpus.

[0067] In contrast, AL sentences can be automatically generated in large quantities. Some embodiments of the present invention employ this insight to increase performance, facilitate training, and shorten the time to market of the transliterator module. A first phase of training can be performed on a relatively large automatically generated AL corpus, thereby producing a partially trained transliterator that can reliably produce grammatically correct AL sentences in the target artificial language. A second phase of training can then be performed on more moderately sized NL-AL corpora.

[0068] Another advantage of two-stage training as described herein is that multiple NL-AL translators can be formed independently of each other without having to repeat the first stage training. Therefore, the decoder part of the translator module can be reused as it is (that is, without the need for repeated training) in multiple translators, which can greatly reduce its development cost and time to market. For example, each of these different translators can correspond to different source natural languages, such as English, German and Chinese. In another example, each different translator can be trained according to different groups of sentences of the same natural language (e.g., English). Such scenarios can occur when each translator is adopted for different tasks / applications (e.g., one translator is used in sales, and another translator is used in database management).

[0069] Although much of the present description is directed to training for automatic translation from natural languages ​​into query languages ​​such as SQL, skilled artisans will appreciate that the described systems and methods may be applicable to other applications and artificial languages. Alternative query languages ​​include SPARQL and other Resource Description Format (RDF) query languages. Exemplary applications of such translations include facilitating access to data represented in RDF, such as for querying from the World Wide Web and / or heterogeneous knowledge bases (e.g., ) Extract information. Another exemplary application of some embodiments of the present invention is to automatically generate RDF code, for example to automatically organize / structure heterogeneous data, or to import data into a database without having expertise in programming or query languages.

[0070] Applications of some embodiments where the target language is a programming language include, among other things, automatically generating code that enables human-computer interaction (e.g., shell scripts). For example, a user may request a machine to perform an action (e.g., call a phone number, access a web page, extract an object, etc.). The automatic translator may translate the corresponding NL command into a set of computer-readable instructions that can be executed in response to receiving the user's command. Other exemplary applications include automatically translating code between two different programming languages ​​(e.g., Python to ), and automatically process rich media (e.g., annotate images and videos).

[0071] In yet another example, some computer security vendors that specialize in detecting malware use a proprietary programming language (a version of bytecode) to encode malware detection routines and / or malware indicative signatures. Some embodiments of the present invention may facilitate such anti-malware research and development by allowing an operator to automatically generate bytecode routines without having specialized knowledge of bytecode.

[0072] It will be apparent to those skilled in the art that the above embodiments can be modified in many ways without departing from the scope of the invention. Therefore, the scope of the invention should be determined by the appended claims and their legal equivalents.

Claims

1. A method comprising: using at least one hardware processor of a computer system to Training an automatic natural language NL to artificial language AL translator, wherein training the NL to AL translator comprises: Execute the first phase of training; Determining whether a first-stage training termination condition is met according to at least one performance criterion in the first-stage training; and In response to satisfying the first stage training termination condition, performing the second stage training; The execution of the first stage of training includes: executing an AL encoder configured to receive a first input array comprising a representation of an input AL sentence expressed in an artificial language and in response generate a first internal array, and a decoder coupled to the AL encoder configured to receive the first internal array and in response generate a first output array comprising a representation of a first output AL sentence expressed in the artificial language; determining a first similarity score indicative of a similarity between the input AL sentence and the first output AL sentence in response to providing the first input array to the AL encoder; and adjusting a first set of parameters of the decoder based on the first similarity score to improve the match between the AL encoder input and the decoder output; And wherein performing the second stage training comprises: executing an NL encoder configured to receive a second input array comprising a representation of an input NL sentence expressed in a natural language and in response output a second internal array to the decoder; determining a second output array produced by the decoder in response to receiving the second internal array, the second output array comprising a representation of a second output AL sentence expressed in the artificial language; determining a second similarity score indicating a degree of similarity between the second output AL sentence and a target AL sentence, the target AL sentence comprising a translation of the input NL sentence into the artificial language; and A second set of parameters of the NL encoder is adjusted according to the second similarity score to improve a match between a decoder output and a target output representing a corresponding translation of an input received by the NL encoder into the artificial language.

2. The method of claim 1, wherein the artificial language comprises a term selected from the group consisting of a database query language, a programming language, and a markup language.

3. The method according to claim 2, wherein the artificial language is Structured Query Language (SQL).

4. The method of claim 1, further comprising determining the second similarity score based on a similarity between the second output array and a target array comprising a representation of the target AL sentence.

5. The method of claim 1 , wherein determining the second similarity score comprises employing at least one hardware processor of the computer system to: Inputting the representation of the target AL sentence into the AL encoder; determining a third internal array comprising the output of the AL encoder in response to receiving the representation of the target AL sentence as input; and The second similarity score is determined according to a similarity between the second internal array and the third internal array.

6. The method of claim 1 , further comprising, in response to determining whether the first phase training termination condition is satisfied, employing at least one hardware processor system of the computer system to: providing a third input array to the AL encoder, the third input array comprising a representation of a third AL sentence expressed in the artificial language; determining a third output array comprising an output of the decoder in response to the AL encoder receiving the third input array, the third output array comprising a representation of a third output AL sentence; determining a third similarity score indicating a degree of similarity between a third input AL sentence and said third output AL sentence; The first set of parameters of the decoder is adjusted based on the third similarity score to improve the match between the AL encoder input and the decoder output.

7. The method of claim 1, further comprising, in response to performing the second phase of training, employing at least one hardware processor of the computer system to: executing another NL encoder configured to receive a third input array comprising a representation of another NL sentence, and in response output a third internal array to the decoder, wherein the another NL sentence is expressed in another natural language different from the natural language; determining a third output array produced by the decoder in response to receiving the third internal array, the third output array comprising a representation of a third output AL sentence; determining a third similarity score indicating a degree of similarity between the third output AL sentence and a third target AL sentence, the third target AL sentence comprising a translation of the other NL sentence into the artificial language; and A third set of parameters of the other NL encoder is adjusted according to the third similarity score to improve a match between decoder outputs and another set of target outputs representing respective translations of inputs received by the other NL decoder into the artificial language.

8. The method of claim 1, wherein the NL encoder comprises a recurrent neural network.

9. A computer system comprising at least one hardware processor and a memory, wherein the at least one hardware processor is configured to: Training an automatic natural language NL to artificial language AL translator, wherein training the NL to AL translator comprises: Execute the first phase of training; Determining whether a first-stage training termination condition is met according to at least one performance criterion in the first-stage training; and In response to satisfying the first stage training termination condition, performing the second stage training; The execution of the first stage of training includes: executing an AL encoder configured to receive a first input array comprising a representation of an input AL sentence expressed in an artificial language and in response generate a first internal array, and a decoder coupled to the AL encoder configured to receive the first internal array and in response generate a first output array comprising a representation of a first output AL sentence expressed in the artificial language; determining a first similarity score indicative of a similarity between the input AL sentence and the first output AL sentence in response to providing the first input array to the AL encoder; and adjusting a first set of parameters of the decoder based on the first similarity score to improve the match between the AL encoder input and the decoder output; And wherein performing the second stage training comprises: executing an NL encoder configured to receive a second input array comprising a representation of an input NL sentence expressed in a natural language and in response output a second internal array to the decoder; determining a second output array produced by the decoder in response to receiving the second internal array, the second output array comprising a representation of a second output AL sentence expressed in the artificial language; determining a second similarity score indicating a degree of similarity between the second output AL sentence and a target AL sentence, the target AL sentence comprising a translation of the input NL sentence into the artificial language; and A second set of parameters of the NL encoder is adjusted according to the second similarity score to improve a match between a decoder output and a target output representing a corresponding translation of an input received by the NL encoder into the artificial language.

10. The computer system of claim 9, wherein the artificial language comprises a term selected from the group consisting of a database query language, a programming language, and a markup language.

11. The computer system of claim 10, wherein the artificial language is Structured Query Language (SQL).

12. The computer system of claim 9, wherein the at least one hardware processor is further configured to determine the second similarity score based on a similarity between the second output array and a target array comprising a representation of the target AL sentence.

13. The computer system of claim 9, wherein determining the second similarity score comprises employing the at least one hardware processor of the computer system to: Inputting the representation of the target AL sentence into the AL encoder; determining a third internal array comprising the output of the AL encoder in response to receiving the representation of the target AL sentence as input; and The second similarity score is determined according to a similarity between the second internal array and the third internal array.

14. The computer system of claim 9, wherein the at least one hardware processor is further configured, in response to determining whether a first termination condition is satisfied, if the first termination condition is not satisfied: providing a third input array to the AL encoder, the third input array comprising a representation of a third AL sentence expressed in the artificial language; determining a third output array comprising an output of the decoder in response to the AL encoder receiving the third input array, the third output array comprising a representation of a third output AL sentence; determining a third similarity score indicating a degree of similarity between a third input AL sentence and said third output AL sentence; The first set of parameters of the decoder is adjusted based on the third similarity score to improve the match between the AL encoder input and the decoder output.

15. The computer system of claim 9, wherein the at least one hardware processor is further configured to, in response to performing the second phase training: executing another NL encoder configured to receive a third input array and in response output a third internal array to the decoder, wherein another NL sentence is expressed in another natural language different from the natural language; determining a third output array produced by the decoder in response to receiving the third internal array, the third output array comprising a representation of a third output AL sentence; determining a third similarity score indicating a degree of similarity between the third output AL sentence and a third target AL sentence, the third target AL sentence comprising a translation of the other NL sentence into the artificial language; and A third set of parameters of the other NL encoder is adjusted according to the third similarity score to improve a match between decoder outputs and another set of target outputs representing respective translations of inputs received by the other NL decoder into the artificial language.

16. The computer system of claim 9, wherein the NL encoder comprises a recurrent neural network.

17. A non-transitory computer-readable medium storing instructions that, when executed by a first hardware processor of a first computer system, cause the first computer system to form a trained natural language (NL) to artificial language (AL) translator module comprising an NL encoder and a decoder connected to the NL encoder, wherein training the NL to AL translator module comprises employing a second hardware processor of a second computer system to: Execute the first phase of training; Determining whether a first-stage training termination condition is met according to at least one performance criterion in the first-stage training; and In response to satisfying the first stage training termination condition, performing the second stage training; The execution of the first stage of training includes: coupling the decoder to an artificial language AL encoder, the AL encoder configured to receive a first input array comprising a representation of an input AL sentence expressed in an artificial language and in response generate a first internal array, the AL encoder coupled to the decoder such that the decoder receives the first internal array and in response generates a first output array comprising a representation of a first output AL sentence expressed in the artificial language; determining a first similarity score indicative of a similarity between the input AL sentence and the first output AL sentence in response to providing the first input array to the AL encoder; and adjusting a first set of parameters of the decoder based on the first similarity score to improve the match between the AL encoder input and the decoder output; And wherein performing the second stage training comprises: coupling the NL encoder to the decoder such that the NL encoder receives a second input array comprising a representation of an input NL sentence expressed in a natural language and in response outputs a second internal array to the decoder; determining a second output array produced by the decoder in response to receiving the second internal array, the second output array comprising a representation of a second output AL sentence expressed in the artificial language; determining a second similarity score indicating a degree of similarity between the second output AL sentence and a target AL sentence, the target AL sentence comprising a translation of the input NL sentence into the artificial language; and A second set of parameters of the NL encoder is adjusted according to the second similarity score to improve a match between a decoder output and a target output representing a corresponding translation of an input received by the NL encoder into the artificial language.

18. A computer system comprising a first hardware processor configured to execute a trained natural language (NL) to artificial language (AL) translator module comprising an NL encoder and a decoder connected to the NL encoder, wherein training the NL to AL translator module comprises employing a second hardware processor of a second computer system to: Execute the first phase of training; Determining whether a first-stage training termination condition is met according to at least one performance criterion in the first-stage training; and In response to satisfying the first stage training termination condition, performing the second stage training; The execution of the first stage of training includes: coupling the decoder to an AL encoder configured to receive a first input array comprising a representation of an input AL sentence expressed in an artificial language and in response generate a first internal array, the AL encoder coupled to the decoder such that the decoder receives the first internal array and in response generates a first output array comprising a representation of a first output AL sentence expressed in the artificial language; determining a first similarity score indicative of a similarity between the input AL sentence and the first output AL sentence in response to providing the first input array to the AL encoder; and adjusting a first set of parameters of the decoder based on the first similarity score to improve the match between the AL encoder input and the decoder output; And wherein performing the second stage training comprises: coupling the NL encoder to the decoder such that the NL encoder receives a second input array comprising a representation of an input NL sentence expressed in a natural language and in response outputs a second internal array to the decoder; determining a second output array produced by the decoder in response to receiving the second internal array, the second output array comprising a representation of a second output AL sentence expressed in the artificial language; determining a second similarity score indicating a degree of similarity between the second output AL sentence and a target AL sentence, the target AL sentence comprising a translation of the input NL sentence into the artificial language; and A second set of parameters of the NL encoder is adjusted according to the second similarity score to improve a match between a decoder output and a target output representing a corresponding translation of an input received by the NL encoder into the artificial language.

Images