Method and apparatus for training a sequence generation model

Abstract

The embodiment of the specification provides a method and device for training a sequence generation model. According to the method, first, an original training sample is obtained, which includes a first sentence for querying a target data table through a natural language, metadata of the target data table, and a second sentence as a data query language. Then, according to a preset probability, a predetermined number of noise adding operations are applied to the training sample, wherein any one of the noise adding operations at least includes modifying one of the first sentence and the second sentence, thereby generating a noise-added sample including a source sentence, target metadata and a target sentence. Then, an input sequence is formed based on the source sentence and the target metadata, the input sequence is processed by using the sequence generation model to obtain an output sequence, and the sequence generation model is updated according to the output sequence and the target sentence.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to database query and artificial intelligence, and in particular, to a method and apparatus for training a sequence generation model for generating a query statement. BACKGROUND

[0002] It is known that a query to a database generally needs to use a data query language specific to the database. For example, for a relational database, SQL language is a dedicated structured data query language. However, this brings some query difficulties to users who are not familiar with the professional data query language.

[0003] Text-to-SQL is a technology of converting natural language to SQL, which belongs to a kind of semantic parsing. Text-to-SQL technology can enable users without relevant data mining professional background to directly interact with the database through natural language questioning, query and analyze the results, thereby reducing the threshold of data analysis. Since the process of natural language→SQL→query result has better explainability than the table-based question and answer scheme of directly returning results according to the question, the technology can have good prospects in the table-based question and answer scene.

[0004] At present, it has been proposed to use a machine learning model to realize the above conversion of natural language to data query language. In the scheme of the machine learning model, the data query language is usually predicted as an output with a certain probability. Since there is a very high requirement for the accuracy of the converted data query language in the data query scene, it is urgent to improve the prediction accuracy of the machine learning model. SUMMARY

[0005] In view of the above problems, the present specification provides a method for training a sequence generation model, which enhances the applicability and robustness of the sequence generation model to the data query language through a noise-adding and noise-removing learning manner, thereby improving the performance of the conversion of natural language to data query language.

[0006] According to a first aspect, a method for training a sequence generation model is provided, comprising:

[0007] obtaining a training sample, wherein the training sample comprises: a first statement for querying a target data table through a natural language, original metadata, and a second statement as a data query language; the original metadata indicates field information of the target data table;

[0008] applying a predetermined number of noise-adding operations on the training sample according to a preset probability, wherein any noise-adding operation at least comprises modifying one of the first statement and the second statement;

[0009] generate a noisy sample according to a result of the noise adding operation, wherein the noisy sample includes the source sentence, the target metadata, and the target sentence;

[0010] form an input sequence based on the source sentence and the target metadata, process the input sequence by using the sequence generation model to obtain an output sequence;

[0011] update the sequence generation model according to the output sequence and the target sentence.

[0012] According to an embodiment, the predetermined plurality of noise adding operations applied to the training sample include: applying a first noise adding operation to the training sample according to a first probability, the first noise adding operation including:

[0013] performing a first modification on the original metadata to obtain target metadata;

[0014] modifying the second sentence according to the first modification;

[0015] generating a noisy sample according to a result of the noise adding operation includes taking the first sentence as the source sentence and taking the modified second sentence as the target sentence.

[0016] In one embodiment of the above embodiment, the original metadata includes a plurality of data items spliced in sequence, and a single data item includes a field sequence number and a field name corresponding to a field of the target data table; the second sentence is obtained by replacing the field name in the original data query sentence with the corresponding field sequence number. In this case, the first modification can include: scrambling the field sequence numbers in the plurality of data items; and the corresponding modification of the second sentence includes modifying the field sequence numbers in the second sentence according to the scrambled field sequence numbers.

[0017] In another embodiment of the above embodiment, the first modification includes: randomly deleting field information of a plurality of fields from the original metadata; and the corresponding modification of the second sentence includes: replacing symbols in the second sentence that refer to the plurality of fields with a first predetermined symbol representing an exception.

[0018] In yet another embodiment of the above embodiment, the first modification includes: inserting a new field from another data table into the original metadata; and the corresponding modification of the second sentence includes: modifying the field sequence numbers in the second sentence according to the field sequence numbers after the insertion of the new field.

[0019] According to an embodiment, the predetermined plurality of noise adding operations applied to the training sample include: applying a second noise adding operation to the training sample according to a second probability, the second noise adding operation including:

[0020] select one of the first sentence and the second sentence as a selected sentence according to a preset selection probability;

[0021] modify the selected sentence to obtain a modified sentence;

[0022] generating the noisy sample according to the result of the noisy operation includes taking the modified sentence as the source sentence, taking the original metadata as the target metadata, and taking the selected sentence as the target sentence.

[0023] In one of the above embodiments, the modifying of the selected sentence to obtain a modified sentence includes: obtaining a plurality of entity words in the selected sentence; and scrambling the order of the plurality of entity words in the sentence.

[0024] According to an embodiment, the input sequence is formed based on the source sentence and the target metadata, and specifically includes: determining a type identifier according to whether the target sentence is a natural language or a data query language; and adding the type identifier to the source sentence and the target metadata to obtain the input sequence.

[0025] According to the above embodiment, processing the input sequence by using the sequence generation model to obtain an output sequence can include: when the type identifier takes a first identifier indicating a natural language, the sequence generation model performs first processing to obtain an output sequence in a natural language form; and when the type identifier takes a second identifier indicating a data query language, the sequence generation model performs second processing to obtain an output sequence in a data query language form.

[0026] In one embodiment, the sequence generation model includes a pointer generation network; and processing the input sequence by using the sequence generation model to obtain an output sequence specifically includes: for a current position of the output sequence to be output, the pointer generation network determines, according to the encoding of the input sequence, a first probability distribution of selecting each vocabulary in a target vocabulary set and a second probability distribution of copying each word in the input sequence, and determines an output vocabulary of the current position according to the first probability distribution and the second probability distribution.

[0027] According to an implementation, the predetermined several noisy operations are applied to the training sample, including: inputting the training sample into several noisy modules, the several noisy modules are respectively used to perform the several noisy operations, and the activation probabilities of the several noisy modules respectively conform to the preset probabilities.

[0028] According to a second aspect, a device for training a sequence generation model is provided, including:

[0029] The training sample acquisition unit is configured to acquire training samples, which include a first sentence for querying a target data table through a natural language, original metadata, and a second sentence as a data query language; the original metadata indicates field information of the target data table;

[0030] The noise adding operation unit is configured to add predetermined noise operations to the training samples according to a preset probability, wherein any noise operation at least includes modifying one of the first sentence and the second sentence;

[0031] The noise-added sample generation unit is configured to generate noise-added samples according to noise operation results, wherein the noise-added samples include a source sentence, target metadata, and a target sentence;

[0032] The sequence processing unit is configured to form an input sequence based on the source sentence and the target metadata, process the input sequence by using the sequence generation model, and obtain an output sequence;

[0033] The updating unit is configured to update the sequence generation model according to the output sequence and the target sentence.

[0034] According to a third aspect, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed in a computer, the computer program causes the computer to execute the method according to the first aspect.

[0035] According to a fourth aspect, a computing device is provided, which includes a memory and a processor. The memory stores executable code, and the processor executes the executable code to implement the method according to the first aspect.

[0036] In the embodiments of the present disclosure, the original training samples are subjected to noise operation with a certain probability, and the sequence generation model is trained by using the noise-added samples thus obtained. The noise-added samples subjected to the erosion operation can assist the sequence generation model to better recognize the relevance between the query target and the metadata, and the noise-added samples subjected to the cutting and washing operation can assist the sequence generation model to better learn the relationship between entities and entities, and between entities and their components. Therefore, the above training process constructs noise-added samples in view of the particularity of the data query language, and enhances the applicability and robustness of the sequence generation model for conversion of the data query language, thereby improving the conversion performance of the data query language. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0038] Figure 1 An illustration of language conversion using a sequence generation model;

[0039] Figure 2 A flowchart of a method of training a sequence generation model according to one embodiment;

[0040] Figure 3 An illustration of one specific example of a training sample;

[0041] Figure 4 An illustration of one specific example of a training sample after erosion and noise operation;

[0042] Figure 5 An illustration of one specific example of a training sample after cut and paste and noise operation;

[0043] Figure 6 An illustration of an apparatus for training a sequence generation model according to one embodiment. DETAILED DESCRIPTION

[0044] The schemes provided in the present specification will be described below with reference to the accompanying drawings.

[0045] In order to facilitate database query operations for users who are not familiar with data query language, a model using machine learning is proposed to realize conversion from natural language to data query language. In such a case, the machine learning model essentially realizes a sequence-to-sequence (seq2seq) conversion task of converting an input sequence in one language form to an output sequence in another language form. Therefore, a mainstream sequence generation model using an encoding-decoding architecture can be used to perform the seq2seq generation task.

[0046] Figure 1 An illustration of language conversion using a sequence generation model. As shown in Figure 1 The sequence generation model can include an encoder and a decoder. The encoder obtains an input sequence in language A, encodes it to obtain an encoded hidden vector H. The decoder performs decoding operation according to the encoded hidden vector H, and outputs an output sequence in language B character by character. When language A and language B are different natural languages, the above process can realize machine translation of different languages. When language A is a natural language and language B is a data query language such as SQL, the above process is equivalent to the task of converting a natural language query statement into a data query language.

[0047] Although it belongs to a sequence-to-sequence task, the conversion task from natural language to data query language (such as SQL) has its particularity compared to ordinary sequence conversion tasks (such as translation between two natural languages).

[0048] The first particularity comes from the asymmetry of the language space. It can be understood that in the traditional machine translation problem, the source language and the target language are both natural languages, and have similar expression completeness. This means that the expression of the source language can mostly find similar correspondence in the target language. While the data query language such as SQL is a computer execution language specially designed for data query, which has a large gap in expression ability with natural language, which makes the translation process from natural language to SQL involve a large number of "many to one" cases. At the same time, there are a large number of order-independent components in the SQL language (for example, the conditions in the WHERE clause), so that the one-to-one training task is difficult to represent this grammatical feature of SQL.

[0049] The particularity of the other aspect comes from the need to be integrated with database information. In traditional machine translation, both the input and the output are natural language sequences, but in the task of converting from natural language to data query language, the metadata information (schema) of the database / data table to be queried needs to be referred to, so the metadata is also input into the model. The SQL statement corresponding to the same natural language question, even if the way of asking is basically the same, will change due to the difference in metadata, so that the corresponding SQL has a completely different expression form.

[0050] Due to the above particularities, especially some order-independent properties of SQL statements (such as condition columns), the end-to-end sequence generation of the sequence generation model shown in the figure brings some noise, so that the prediction performance needs to be improved. Figure 1

[0051] In a related scheme, a multi-task based slot value filling scheme is also proposed according to the characteristics of the SQL language. Specifically, a general SQL template is first constructed, and a complete SQL statement is split into multiple clauses such as SELECT, WHERE, etc. through the template, and each clause has some fillable slots. Thus, the task of generating a SQL statement is converted into filling the corresponding slot values of each clause. In this way, multiple sub-tasks are established to capture the slot values of different SQL clauses, which can simplify the Text-to-SQL task. This scheme usually has better maintainability than direct sequence generation, because each component of the prediction result is relatively independent and can be split, and when a prediction error occurs, it can also be quickly traced back. However, the need to define a general template in advance leads to the need to define the type of SQL syntax that can be generated at the beginning. If new and more complex sentence patterns need to be supported later, or other data query languages similar to it but with slightly different syntax, the sub-tasks need to be redeveloped. At the same time, as the number of tasks increases, the cascading relationship between sub-tasks will become more complex, which will limit the scalability of the model.

[0052] ​Based on the above research and analysis of the inventor, an improved scheme is proposed in the present specification, considering the above-mentioned speciality of data query language, from the perspective and direction of noise adding and denoising learning, enhancing the applicability and robustness of seq2seq sequence generation model, so as to improve the conversion performance of data query language. The implementation of the improved scheme is described in detail below.

[0053] Figure 2 A method flowchart for training a sequence generation model according to one embodiment is shown. The method can be executed by any computing unit, platform, server, device, etc. with computing and processing capabilities. As shown, the training method can include the steps described in detail below. Figure 2

[0054] First, in step 21, a training sample is obtained, which includes: a first sentence for querying a target data table through natural language, metadata of the above-mentioned target data table, and a second sentence as a data query language. Hereinafter, the training sample is denoted as {Q, S, Y} in some examples, where Q is a natural language sentence, i.e. the above-mentioned first sentence; S is metadata indicating field information of the target data table; Y is a data query sentence, i.e. the above-mentioned second sentence. It can be understood that the training target of the sequence generation model is that when the first sentence Q and the metadata S are concatenated together and input into the model, the model can output the second sentence Y as the converted sentence.

[0055] In one embodiment, each part of the training sample can be optimized in format to better assist model training.

[0056] Specifically, the field information of each field of the target data table T to be queried can be serialized, thereby obtaining the serialized metadata S. In one embodiment, the metadata S includes a plurality of data items concatenated in sequence, each data item corresponding to each field (or each column) of the target data table in sequence. A single data item includes a field serial number and a field name corresponding to one field of the target data table.

[0057] In one specific example, the information of each column in the target data table can be represented by a tuple, and the tuple can include the field serial number and the field name corresponding to the column. <coli>A template such as [col name]:[col type] collates each data item, and the sequence of the spliced data items can be used as the metadata S. In the above template, the prefix " <coli>"represents the i-th column in the data table, i.e. the field sequence number, "col name" is the field name, and "col type" is the field type indicating the value type of the field, such as numerical type, string type, etc.

[0058] For the second statement Y, as a structured query statement for querying the target data table T, the field information in the target data table T is generally mentioned. The field name in the original query statement can be replaced by the corresponding field sequence number according to the sequence number of each field in the above metadata S, as a data query statement for model training. Further, in an embodiment, the field name, field value and condition value mentioned in the data query statement are also marked with special symbols, for example, surrounded by "`" symbols, to assist the model in understanding.

[0059] Figure 3 A specific example of training samples is shown. In this example, the target data table table contains multiple fields such as week, data, opponent, result, attendance, etc. For such a data table, the sequence of its metadata S can contain multiple data items arranged in sequence and spliced: <col0>week <col1>data <col2>opponent, etc. Among them, for simplicity, the field type col type information is not shown. The natural language sentence Q for querying the data table is, for example, which week had an attendance of 53677. The original SQL sentence corresponding to the natural language sentence is, for example, SELECT week from table Where attendance = 53677. It is to be understood that the natural language sentence and the corresponding original SQL sentence can be generated by manual writing or obtained by labeling based on historical query records. By replacing the field name in the above original SQL sentence with the corresponding field serial number in the metadata S, specifically, replacing week with <col0>replace attendance with <col4>and adding special symbols, the statement shown in the figure can be obtained: SELECT <col0>from table Where <col4>`=` 53677 `.` This statement can be taken as the second statement Y in the training sample. Figure 3 In the example shown in FIG. 13, the training objective of the sequence generation model is to, when inputting the metadata S and the natural language statement Q as the input sequence into the sequence generation model, obtain the second statement Y shown in the figure as the output sequence.

[0060] Based on the training sample in the above form, next, in step 22, according to a preset probability, a predetermined number of noise adding operations are applied to the above training sample, wherein any one of the noise adding operations at least includes modifying one of the first statement and the second statement. Then, in step 23, a noise-added sample is generated according to the noise adding operation result, which includes the source statement, the target metadata and the target statement.

[0061] According to an embodiment of the present specification, a number of noise adding operations are preset and a probability of applying is set. For the obtained training sample, the noise adding operation is applied according to the corresponding application probability of each noise adding operation. The specific noise adding process of a number of noise adding operations is described below.

[0062] According to an embodiment, the number of noise adding operations preset includes a first noise adding operation, which is also referred to as "erosion" operation hereinafter. The erosion operation enhances the recognition ability of the sequence generation model for the metadata correlation and the robustness to noise by adding noise in the metadata of the training sample and correspondingly modifying the second statement.

[0063] Specifically, in an embodiment, the erosion operation can include a rearrangement sub-operation, in which the field sequence numbers in a plurality of data items contained in the metadata are shuffled or rearranged, that is, the corresponding relationship between the field sequence number and the field name is shuffled. Since the metadata S is modified, the second statement needs to be correspondingly modified to be consistent with the information recorded in the metadata S. In the case of the rearrangement sub-operation, the field sequence number in the second statement Y needs to be modified according to the shuffled field sequence number.

[0064] In an embodiment, the erosion operation can also include a removal sub-operation, in which the field information of a number of fields is randomly deleted from the metadata. It needs to be understood that if the field to be queried is not contained in the metadata (meaning that the information of the field is not contained in the data table), the field cannot be queried because the information of the data table is insufficient, at which time an exception should be thrown to indicate that a feasible data query statement cannot be constructed. Therefore, for the case of the removal sub-operation, the corresponding modification of the second statement can include that if a number of fields removed are referenced in the second statement, the symbols of these fields (originally the corresponding field sequence numbers) are replaced with a predetermined symbol indicating an exception, such as unk (indicating unknown) or exp (indicating exception).

[0065] In another embodiment, the erosion operation may further include an add sub-operation, in which some fields from other data tables are randomly inserted into the metadata of the target data table. Generally, inserting fields from other data tables will also change the original field numbers in the metadata. In this case, the field numbers in the second statement Y need to be modified according to the changed field numbers.

[0066] In one implementation, when a training sample is selected for erosion with a certain probability P1, the aforementioned rearrangement, removal, and addition sub-operations can be performed sequentially on that training sample. In another implementation, a certain sub-probability can be set for each of the rearrangement, removal, and addition sub-operations. The training sample is then subjected to one or more combinations of these sub-operations based on their respective sub-probabilities.

[0067] Figure 4 This example illustrates a specific instance of a training sample that has undergone an erosion and noise-adding operation. The sample in this example is... Figure 3 It is obtained by performing erosion and noise addition operations based on the training samples. Specifically, based on Figure 3 The original metadata was subjected to a rearrangement sub-operation, for example, changing the field number of the "attendance" field from its original value. <col4>rearranged to <col0>, a remove sub-operation is applied, for example, the field "week" is removed from the original metadata; and an add sub-operation, in which the field "venue" of the other data table is added in the original metadata. Thus, the modified metadata S' is obtained. Correspondingly, for the field "attendance" whose field serial number has changed, the symbol representing the field in the SQL statement is modified to the rearranged field serial number <col0>"unk" for the removed field "week". Thus, the modified second statement Y' is obtained.

[0068] In the process of the erosion operation implemented in the above various sub-operation manners, the metadata is first perturbed and modified (noise is added), and then the second statement is modified accordingly, while the first statement remains unchanged. Correspondingly, in the noisy sample {Ts, S', Tt} generated based on the erosion operation, the source statement Ts is the original first statement Q, the target metadata S' is the perturbed and modified metadata, and the target statement Tt is the statement obtained by modifying the second statement Y accordingly. The noisy sample thus obtained, since the input (S) and output (Y) are jointly modified, can train the sequence generation model to identify the fields in the target data table that are truly relevant to the current query, and learn to "throw an exception" when the information in the table is insufficient to generate a complete data query statement, thereby enhancing the sequence generation model's ability to identify the relevance of the metadata.

[0069] According to an embodiment, the pre-set plurality of noise addition operations further includes a second noise addition operation, which is referred to as "cutting and washing" operation hereinafter. Unlike the erosion operation which modifies both the metadata (represented in natural language) and the data query language, the cutting and washing operation only acts on a single sequence in a single language space, perturbs and modifies the single sequence, and trains the sequence generation model to restore the original sequence without perturbation.

[0070] Specifically, in one embodiment, the cutting and washing operation can include selecting one of the first statement and the second statement of the training sample as a selected statement according to a pre-set selection probability, and then modifying the selected statement to obtain a modified statement. The modification of the selected statement can be implemented in various ways, such as rearranging a plurality of words in the selected statement, deleting words, adding irrelevant words, etc. In one implementation, entity words can be identified from the selected statement, including table names, field names, field values, etc. Then, the entity words are shuffled to disrupt their order in the selected statement, thus obtaining the modified statement.

[0071] Figure 5 A specific example of a training sample subjected to the cutting and washing noise addition operation is shown. The sample in this example is based on the training sample Figure 3 and is obtained by performing the cutting and washing operation thereon. Specifically, for the training sample Figure 3 The original natural query sentence Q: "which week had an attendance of 53677" is subjected to a shuffling operation that disrupts the order of entity words, resulting in the modified natural query sentence Q': "which 53677 had an week of attendance". As an example, for the original data query sentence Y: "select * from table where column = 53677", a shuffling operation is also applied that disrupts the order of words, resulting in the modified data query sentence Y' as shown in Figure 3 Figure 5 It is noted that although the modifications to the natural query sentence Q and the data query sentence Y are shown in Figure 5

[0072] In the above shuffling operation, the metadata remains unchanged, and one of the first sentence and the second sentence is selected as the selected sentence that is subjected to the perturbation modification. In this way, in the generated noisy sample {Ts, S', Tt}, the source sentence Ts is the modified sentence, the target metadata S' is the original metadata S, and the target sentence Tt is the selected sentence. In this way, the sequence generation model is trained to predict the unmodified selected sentence based on the metadata and the modified sentence. When the selected sentence is the first sentence Q, the sequence generation model is trained to restore the original natural query sentence Q based on the metadata and the modified natural query sentence Q'. When the selected sentence is the second sentence Y, the sequence generation model is trained to restore the original data query sentence Y based on the metadata and the modified data query sentence Y'. Through the learning of shuffling and denoising, the sequence generation model can learn the association between various entity words, especially the relationship between entity features and their components. For example, a numerical value is more likely to be a conditional value, and a simple noun is more likely to appear in a conditional clause.

[0073] The above erosion operation and shuffling operation are each applied with a preset probability. Therefore, there is a certain probability that both erosion and shuffling operations are applied to a certain training sample. Since the erosion operation does not change the order and role of each element Q, S, Y in the sample, in an embodiment, when it is determined that both noise operations are applied, the erosion operation can be set to be applied first, and the sample after the erosion operation is used as the input sample of the shuffling operation, and the shuffling operation is continued to be applied.

[0074] ​​According to an implementation, the operation algorithms of the erosion operation and the cut-paste operation can be packaged as modules to obtain a plurality of noise adding modules including the erosion module and the cut-paste module, and the activation probability of each noise adding module is set to be consistent with the aforementioned application probability of the corresponding noise adding operation. Thus, in step 22, the training samples are sequentially input into the noise adding modules, and each noise adding module performs the corresponding noise adding operation when activated. More specifically, for the erosion module, a plurality of sub-modules can be set therein, including a rearrangement sub-module, a removal sub-module and an addition sub-module. When the erosion module is activated, each sub-module applies the corresponding sub-operation with a respective sub-activation probability.

[0075] In an implementation, the activation probability of each noise module, or even each sub-module, can be dynamically adjusted, so as to adjust the proportion of different noise samples in the full sample, prevent the model from overfitting to the fixed noise pattern, and make the model have better generalization ability.

[0076] The above describes the specific operation process of the erosion noise adding operation and the cut-paste noise adding operation. On this basis, other noise adding operations can also be combined as alternative noise adding operations, for example, some conventional data enhancement operations. In this way, by applying a plurality of noise adding operations to the training samples with corresponding probabilities, noise added samples different from the original training samples are obtained.

[0077] Then, in step 24, the source sentence and the target metadata in the noise added sample are used to form an input sequence, and the input sequence is processed by using the sequence generation model to obtain an output sequence.

[0078] In an embodiment, the source sentence and the target metadata in the noise added sample are spliced to form an input sequence, which is input into the sequence generation model.

[0079] According to an implementation, in the case that the preset plurality of noise adding operations includes the cut-and-paste operation, the processing mode of the sequence generation model needs to be considered. It can be understood that in the normal mode, the input sequence is formed based on the natural language query statement Q and the metadata S, and is input into the sequence generation model, and the model is expected to output a data query statement Y in SQL language. However, as mentioned above, in the cut-and-paste operation, the first statement Q is selected as the selected statement with a certain probability, and is modified. In this case, the source statement in the noise-added sample is the modified first statement Q', and the target statement is the original first statement Q, and it needs to be noted that the first statement is a natural language query statement. This means that in this case, the sequence generation model is required to output a natural language statement as the prediction of the target statement. Therefore, the sequence generation model needs to output statements in different language forms in different processing modes. In order to indicate the language form that the sequence generation model needs to output according to the noise-added sample, in an embodiment, a type identifier can be determined according to the language form of the target statement in the noise-added sample, and the type identifier is added based on the source statement and the target metadata to obtain the input sequence. In this way, the type identifier is contained in the input sequence, and the type identifier can indicate that the sequence generation model uses the corresponding processing mode to output the required language form.

[0080] In an embodiment, the above-mentioned type identifier is added to the input sequence in the form of a prefix. For example, if the type identifier takes the value "2sql", it indicates that the sequence generation model needs to output an SQL statement; if the type identifier takes the value "2nl", it indicates that the sequence generation model needs to output a natural language statement.

[0081] Correspondingly, when the sequence generation model processes the input sequence, different processing processes are performed according to different values of the type identifier. Specifically, when the type identifier takes a first identifier indicating a natural language, such as "2nl", the sequence generation model performs a first processing to obtain an output sequence in a natural language form; when the type identifier takes a second identifier indicating a data query language, such as "2sql", the sequence generation model performs a second processing to obtain an output sequence in a data query language form. The above-mentioned first processing and second processing are essentially different intermediate results obtained by the model processing the type identifier with different values, and different combinations of other model partial processing results. This has been applied in multilingual natural language translation.

[0082] The above-mentioned sequence generation model can adopt the basic architecture of an encoder-decoder, and the encoder / decoder can be implemented by various neural networks. In an example, the encoder-decoder can be implemented by a neural network based on an attention mechanism, such as a Transformer-based encoder-decoder. In another example, the encoder-decoder can be implemented by a neural network based on time sequence, such as an LSTM neural network or an RNN neural network.

[0083] In one example, the sequence generation model can include a pointer generator network as its decoder. Accordingly, the process of the sequence generation model processing an input sequence to obtain an output sequence can include the following steps. An encoder encodes the input sequence to obtain an encoded hidden vector H. A decoder outputs characters or symbols one by one to form the output sequence. For each current position of the output sequence, the pointer generator network determines a first probability distribution {s1, s2, … s|V|} of selecting each vocabulary from a target vocabulary set V according to the encoded hidden vector H of the input sequence, and a second probability distribution {i1, i2, … i|X|} of copying each vocabulary from the input sequence according to a previously output hidden state h, according to the current hidden state h, and determines an output vocabulary for the current position according to both the first probability distribution and the second probability distribution. |V|} and a second probability distribution {i1, i2, … i |X|} of copying each vocabulary from the input sequence, where |V| is the number of vocabularies in the target vocabulary set, |X| is the number of vocabularies in the input sequence X, and each element in the first probability distribution and the second probability distribution can be a (non-normalized) probability score for the corresponding vocabulary. Then the output vocabulary for the current position can be determined according to both the first probability distribution and the second probability distribution.

[0084] In one example, the target vocabulary set V described above can include a training corpus vocabulary subset Vq, a vocabulary Vc related to the original metadata, and a standard vocabulary Vs in the data query language. Taking SQL language as an example, the standard vocabulary Vs in the data query language can include various SQL available keywords, such as Select, Max, Min, Where, etc.

[0085] Through the above various specific neural networks and processing methods, the sequence generation model processes the input sequence to generate the output sequence.

[0086] Then, at step 25, the sequence generation model is updated according to the output sequence and the target sentence.

[0087] Specifically, the target sentence acts as the expected output or label of this time processing of the sequence generation model. Therefore, various loss function forms can be used, such as cross-entropy loss function, to determine the prediction loss based on the difference between the output sequence and the target sentence; and the model parameters of the sequence generation model are updated to minimize the prediction loss, so as to realize the training of the model.

[0088] In the above process, the original training samples are added with noise with a certain probability, and the sequence generation model is trained using the added noise samples thus obtained. Among them, the added noise samples that have undergone the erosion operation can assist the sequence generation model to better identify the relevance of the query target and the metadata, and the added noise samples that have undergone the cut-washing operation can assist the sequence generation model to better learn the relationship between entities and entities, and entities and their components. Therefore, the above training process constructs added noise samples according to the particularity of the data query language, enhances the applicability and robustness of the sequence generation model for data query language conversion, and thus improves the conversion performance of the data query language.

[0089] It should be understood that in the above description, the data query language is mainly described by taking the sql language as an example. However, the above scheme is not limited to conversion to the sql language, but can be applied to other structured query languages with similar characteristics, such as Gremlin, lambda expression, etc.

[0090] After the sequence generation model is trained using the added noise samples, the trained model can be used for natural language-data query language conversion. During the conversion, the natural language q for querying and the metadata s of the data table to be queried are spliced into an input sequence and input into the sequence generation model. In an embodiment, in the case that the model supports added noise samples generated by the cut-washing operation, a category identifier such as 2sql also needs to be added in the input sequence. In this way, the sequence generation model generates an output sequence in the form of a data query language based on the input sequence, thereby realizing the conversion from natural language to data query language.

[0091] Through the test of the test samples, the conversion performance of the sequence generation model trained by the above added noise and denoising learning is greatly improved.

[0092] According to another aspect, embodiments correspond to the training method shown in Figure 2 The present specification also provides a device for training a sequence generation model, which can be deployed in any computing unit, platform, server, or device with computing and processing capabilities. Figure 6 A schematic diagram of a device for training a sequence generation model according to an embodiment is shown. As Figure 6 shown, the device 600 includes:

[0093] A training sample acquisition unit 61 configured to acquire training samples, including: a first sentence for querying a target data table through a natural language, original metadata, and a second sentence as a data query language; the original metadata indicates the field information of the target data table;

[0094] The noise adding operation unit 62 is configured to apply a predetermined number of noise adding operations on the training sample according to a preset probability, wherein any noise adding operation at least includes modifying one of the first sentence and the second sentence;

[0095] The noise sample generation unit 63 is configured to generate a noise sample according to the noise adding operation result, wherein the noise sample includes the source sentence, the target metadata and the target sentence.

[0096] The sequence processing unit 64 is configured to form an input sequence based on the source sentence and the target metadata, process the input sequence by using the sequence generation model to obtain an output sequence.

[0097] The updating unit 65 is configured to update the sequence generation model according to the output sequence and the target sentence.

[0098] The above-mentioned units of the apparatus can be configured to perform the method steps of Figure 2 , and the specific implementation process can refer to the description of the above-mentioned Figure 2 , and will not be repeated here.

[0099] According to other aspects of the embodiments, a computer readable storage medium is also provided, which stores a computer program, when the computer program is executed in a computer, the computer program causes the computer to perform the training method described in the above-mentioned Figure 2 .

[0100] According to still another aspect of the embodiments, a computing device is also provided, which includes a memory and a processor, the memory stores executable code, and the processor executes the executable code to implement the training method described in the above-mentioned Figure 2 Figure 2 .

[0101] Those skilled in the art should be aware that the functions described in the above-mentioned one or more examples can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, these functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium.

[0102] The above-mentioned specific embodiments further explain the purpose, technical solutions and beneficial effects of the present application, and it should be understood that the above-mentioned specific embodiments are only specific embodiments of the present application and are not used to limit the protection scope of the present application, and any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the present application should be included in the protection scope of the present application. < / coli> < / coli> ​

Claims

1. A method for training a sequence generation model, comprising: obtaining a training sample, wherein the training sample comprises a first sentence for querying a target data table by a natural language, original metadata, and a second sentence as a data query language, and the original metadata indicates field information of the target data table; applying a predetermined number of noise adding operations on the training sample according to a preset probability, wherein any one of the noise adding operations comprises at least modifying one of the first sentence and the second sentence; generating a noise-added sample according to a result of the noise adding operation, wherein the noise-added sample comprises a source sentence, target metadata, and a target sentence; forming an input sequence based on the source sentence and the target metadata, processing the input sequence by using the sequence generation model to obtain an output sequence; and updating the sequence generation model according to the output sequence and the target sentence.

2. The method of claim 1, wherein, The applying of the predetermined number of noise adding operations on the training sample comprises: applying a first noise adding operation on the training sample according to a first probability, wherein the first noise adding operation comprises: performing a first modification on the original metadata to obtain target metadata; correspondingly modifying the second sentence according to the first modification; and generating the noise-added sample according to the result of the noise adding operation, wherein the first sentence is taken as the source sentence, and the modified second sentence is taken as the target sentence.

3. The method of claim 2, wherein, The original metadata comprises a plurality of data items concatenated in sequence, and each data item comprises a field serial number and a field name corresponding to one field of the target data table. The second sentence is obtained by replacing field names in an original data query sentence with corresponding field serial numbers. The first modification comprises: scrambling the field serial numbers in the plurality of data items. The corresponding modification of the second sentence comprises: modifying the field serial numbers in the second sentence according to the scrambled field serial numbers. 4.The method of claim 2, wherein the first modification comprises: randomly deleting field information of a plurality of fields from the original metadata; the corresponding modification of the second sentence comprises: replacing symbols referring to the plurality of fields in the second sentence with a first predetermined symbol representing an exception. 5.The method of claim 2, wherein the first modification comprises: inserting a new field from another data table into the original metadata; the corresponding modification of the second sentence comprises: modifying the field serial numbers in the second sentence according to the field serial numbers after the insertion of the new field.

6. The method of any one of claims 1-5, wherein, The applying of the predetermined number of noise adding operations on the training sample comprises: applying a second noise adding operation on the training sample according to a second probability, wherein the second noise adding operation comprises: selecting one of the first sentence and the second sentence as a selected sentence according to a preset selection probability; modifying the selected sentence to obtain a modified sentence; and generating the noise-added sample according to the result of the noise adding operation, wherein the modified sentence is taken as the source sentence, the original metadata is taken as the target metadata, and the selected sentence is taken as the target sentence.

7. The method of claim 6, wherein, The modifying of the selected sentence to obtain the modified sentence comprises: obtaining a plurality of entity words in the selected sentence; scrambling an order of the plurality of entity words in the sentence.

8. The method of claim 1, wherein, Forming an input sequence based on the source sentence and the target metadata, including: Determining a type identifier according to whether the target sentence is a natural language or a data query language; Adding the type identifier based on the source sentence and the target metadata to obtain the input sequence.

9. The method of claim 8, wherein, Processing the input sequence by using the sequence generation model to obtain an output sequence, including: When the type identifier takes a first identifier indicating a natural language, the sequence generation model performs first processing to obtain an output sequence in a natural language form; When the type identifier takes a second identifier indicating a data query language, the sequence generation model performs second processing to obtain an output sequence in a data query language form.

10. The method of claim 1, wherein, The sequence generation model includes a pointer generation network. Processing the input sequence by using the sequence generation model to obtain an output sequence, including: For a current position of the output sequence, the pointer generation network determines, according to encoding of the input sequence, a first probability distribution of selecting each vocabulary from a target vocabulary set and a second probability distribution of copying each word from the input sequence, and determines an output vocabulary of the current position according to the first probability distribution and the second probability distribution.

11. The method of claim 10, wherein, The target vocabulary set includes training corpus vocabularies, vocabularies related to the original metadata, and standard vocabularies in the data query language.

12. The method of claim 1, wherein, According to a preset probability, applying a predetermined number of noise operations to the training sample, including: Inputting the training sample into a plurality of noise modules, the plurality of noise modules being respectively used to perform the plurality of noise operations, and an activation probability of each of the plurality of noise modules conforming to the preset probability.

13. The method of claim 1, wherein, Updating the sequence generation model according to the output sequence and the target sentence, including: Determining a prediction loss based on a difference between the output sequence and the target sentence; Updating model parameters of the sequence generation model with a prediction loss minimization target.

14. An apparatus for training a sequence generation model, including: A training sample acquisition unit configured to acquire a training sample, including a first sentence for querying a target data table through a natural language, original metadata, and a second sentence as a data query language; the original metadata indicating field information of the target data table; A noise operation unit configured to apply a predetermined number of noise operations to the training sample according to a preset probability, wherein any one of the noise operations at least includes modifying one of the first sentence and the second sentence; A noise sample generation unit configured to generate a noise sample including a source sentence, target metadata, and a target sentence according to a noise operation result; A sequence processing unit configured to form an input sequence based on the source sentence and the target metadata, and process the input sequence by using the sequence generation model to obtain an output sequence; An updating unit configured to update the sequence generation model according to the output sequence and the target sentence.

15. A computer-readable storage medium having a computer program stored thereon, when the computer program is executed in a computer, causing the computer to execute the method of any one of claims 1-13.

16. A computing device comprising a memory and a processor, wherein: The memory stores executable code, and the processor executes the executable code to implement the method in any one of claims 1-13. The memory stores executable code, and the processor executes the executable code to implement the method in any one of claims 1-13. The memory stores executable code, and the processor executes the executable code to implement the method in any one of claims 1-13. The memory stores

Images