Method for establishing pre-trained language model, semantic parsing method and device

Abstract

The embodiment of the present application discloses a method for establishing a pre-trained language model, a semantic parsing method and an apparatus. The technical scheme includes: obtaining first training data, the first training data includes more than one set of multi-round dialogue sample pairs and pattern information of associated tables, the multi-round dialogue sample pairs include multiple sample pairs consisting of natural language text and its corresponding structured query language (SQL) statements in a dialogue; using the first training data to train a pre-trained language model including an embedding layer and a transformation (Transformer) network; during the training process, inputting an input sequence consisting of a natural language text, the context of the natural language text, and pattern information into the pre-trained language model, and performing at least one of a conversation semantic parsing task, a dialogue editing strategy prediction task, a mask prediction task, and a column label prediction task. The present application is applicable to scenarios where structured tabular data needs to be modeled.

Description

Technical Field

[0001] The present application relates to the field of computer application technology, and in particular to a method for establishing a pre-trained language model, a semantic parsing method and a device in the field of artificial intelligence. Background Art

[0002] Currently, intelligent question and answer has been widely used in customer service, marketing, service acquisition and other scenarios. It provides users with efficient and personalized experience as a supplement to GUI (Graphical User Interface), and is even integrated into hardware devices such as smart speakers, smart homes, and smart navigation. TableQA (table question and answer) is one of the forms of intelligent question and answer and is widely used in various scenarios. In TableQA, knowledge is organized in the form of tables. For example, the product information of a merchant is stored in the form of a table. When a user asks about the specific attributes of a product, the question and answer model can convert the natural language entered by the user into SQL (Structured Query Language) statements, and then locate the answer from the table.

[0003] Among them, semantic parsing models such as NL2SQL (Natural Language ToStructured Query Language) are usually used to convert natural language into SQL statements. The semantic parsing model can be implemented based on a pre-trained language model, that is, the vector representation of the natural language text is first extracted through the pre-trained language model, that is, the encoding process, and then the downstream conversion model converts it into SQL statements based on the vector representation of the natural language text. Currently, the more common pre-trained language models such as BERT, ERNIE, GPT-3, etc. are not suitable for scenarios such as TableQA that require modeling of structured table data. Summary of the invention

[0004] In view of this, the present application provides a method for establishing a pre-trained language model, a semantic parsing method and a device, which are suitable for scenarios such as TableQA that require modeling of structured tabular data.

[0005] According to a first aspect, a method for establishing a pre-trained language model is provided, the method comprising:

[0006] Acquire first training data, where the first training data includes more than one set of multi-round dialogue sample pairs and pattern information of an associated table, where the multi-round dialogue sample pairs include multiple sample pairs consisting of natural language texts and their corresponding structured query language SQL statements in a dialogue;

[0007] Using the first training data to train a pre-trained language model including an embedding layer and a transformer network, the pre-trained language model is used to extract a vector representation of an input natural language text; during the training process, the natural language text, the context of the natural language text, and an input sequence consisting of the pattern information are used as inputs of the pre-trained language model to perform at least one of a conversation semantic parsing task, a dialogue editing strategy prediction task, a mask prediction task, and a column label prediction task;

[0008] The conversation semantic parsing task is used to predict the semantic relationship between the input natural language text and its context and the pattern information;

[0009] The dialogue editing strategy prediction task is used to predict the editing strategy adopted by the input natural language text compared with its context;

[0010] The mask prediction task is used to predict the masked character content in the input sequence;

[0011] The column label prediction task is used to predict the column name of the associated table covered by the SQL statement corresponding to the input natural language text.

[0012] As an achievable manner in the embodiment, obtaining the first training data includes:

[0013] Obtain multiple sample pairs consisting of natural language texts and SQL statements;

[0014] Each sample pair is used as a sample pair for the first round of dialogue, and sample pairs for subsequent N rounds of dialogue are generated, where N is a positive integer;

[0015] The sample pairs of the first round of dialogue and the sample pairs of the subsequent N rounds of dialogue constitute a set of multi-round dialogue sample pairs.

[0016] As an achievable manner in the embodiment, obtaining a plurality of sample pairs consisting of natural language texts and SQL statements includes:

[0017] Obtaining an SQL syntax tree, wherein the SQL syntax tree is pre-generated using SQL syntax rules;

[0018] Generate one or more SQL statements using the SQL syntax tree in combination with the sampled table data;

[0019] Based on the pre-configured speech templates, corresponding natural language texts are generated for each SQL statement to obtain multiple sample pairs.

[0020] As an achievable manner in the embodiment, taking each sample pair as a sample pair for a first round of dialogue to generate sample pairs for subsequent N rounds of dialogue includes:

[0021] Using the sample pairs of the first round of dialogue as sample pairs of the current round;

[0022] Determine an editing strategy applicable to the sample pair of the current round from a preset editing strategy set;

[0023] An editing strategy is sampled from the determined editing strategies, and the SQL statements and natural language texts in the sample pairs of the current round are edited using the sampled editing strategy, so as to obtain the edited SQL statements and natural language texts as the sample pairs of the next round of dialogue;

[0024] The sample pairs of the next round of dialogue obtained by editing are used as the sample pairs of the current round, and the operation of determining the editing strategy applicable to the sample pairs of the current round from the preset editing strategy set is executed until the sample pairs of the N rounds of dialogue after the sample pairs of the first round of dialogue are obtained.

[0025] As an achievable manner in the embodiment, the first training data further includes: a semantic dependency graph annotated with respect to the natural language text in each sample pair and the pattern information of the association table, the semantic dependency graph including nodes and edges, the nodes including words in the natural language text and the column names of the association table, and the edges indicating the semantic relationship between the nodes;

[0026] The conversation semantic parsing task is used to predict the semantic dependency graph of the input natural language text and its context and the pattern information of the association table. The training goal of this task is to minimize the difference between the prediction result and the corresponding annotation in the training sample.

[0027] As an achievable manner in the embodiment, the editing strategy includes at least one of the following:

[0028] Add conditions, modify conditions, delete conditions, modify focus, modify aggregate functions, delete focus, restart, refuse recognition and switch tables.

[0029] As an implementable manner in the embodiment, the context of the natural language text includes: historical natural language text of the natural language text in a conversation, or historical SQL statements, or semantic graphs corresponding to historical SQL statements;

[0030] The semantic graph includes nodes and edges, the nodes include operation keywords, condition keywords, focus parameters and condition parameters, and the edges include aggregation functions and condition operators.

[0031] In a second aspect, a method for establishing a semantic parsing model is provided, the method comprising:

[0032] Acquire second training data, where the second training data includes more than one set of multi-round dialogue sample pairs and pattern information of an associated table, where the multi-round dialogue sample pairs include multiple sample pairs consisting of natural language texts and their corresponding structured query language SQL statements in a dialogue;

[0033] The second training data is used to train a semantic parsing model including an encoder and a decoder; during the training process, an input sequence consisting of an input natural language text, a context of the natural language text, and the pattern information is used as an input to the encoder, and the encoder outputs a vector representation of the natural language text; the decoder outputs an SQL statement using the vector representation of the natural language text, and the training goal of the semantic parsing model is to minimize the difference between the output result of the decoder and the SQL statement corresponding to the input natural language text in the second training data;

[0034] Wherein, the encoder adopts the pre-trained language model established by the method provided by the first aspect above.

[0035] As an achievable manner in the embodiment, obtaining the second training data includes:

[0036] One or more multi-round dialogue sample pairs are sampled from the first training data, and manual proofreading results of natural language texts in the sampled multi-round dialogue sample pairs are obtained; and the proofread multi-round dialogue sample pairs are used as the second training data.

[0037] In a third aspect, a semantic parsing method is provided, the method comprising:

[0038] Obtaining the natural language text corresponding to the question input by the user, the context of the natural language text in the same conversation, and the mode information of the associated table;

[0039] Inputting an input sequence consisting of the natural language text, the context of the natural language text, and the pattern information into a semantic parsing model to obtain an SQL statement output by the semantic parsing model;

[0040] The semantic parsing model is pre-established using the method provided in the second aspect.

[0041] As an achievable manner in the embodiment, the method further includes:

[0042] Query the associated table using the SQL statement;

[0043] The query results are used to generate answers corresponding to the questions.

[0044] In a fourth aspect, a device for establishing a pre-trained language model is provided, the device comprising:

[0045] A first data acquisition unit is configured to acquire first training data, wherein the first training data includes more than one set of multi-round dialogue sample pairs and pattern information of an associated table, wherein the multi-round dialogue sample pairs include multiple sample pairs consisting of natural language texts and their corresponding structured query language SQL statements in a dialogue;

[0046] A first model training unit is configured to use the first training data to train a pre-trained language model including an embedding layer and a transfomer network, wherein the pre-trained language model is used to extract a vector representation of an input natural language text; during the training process, the natural language text, the context of the natural language text, and an input sequence consisting of the pattern information are used as inputs of the pre-trained language model to perform at least one of a conversation semantic parsing task, a dialogue editing strategy prediction task, a mask prediction task, and a column label prediction task;

[0047] The conversation semantic parsing task is used to predict the semantic relationship between the input natural language text and its context and the pattern information;

[0048] The dialogue editing strategy prediction task is used to predict the editing strategy adopted by the input natural language text compared with its context;

[0049] The mask prediction task is used to predict the masked character content in the input sequence;

[0050] The column label prediction task is used to predict the column labels of the associated table covered by the SQL statement corresponding to the input natural language text.

[0051] In a fifth aspect, a device for establishing a semantic parsing model is provided, the device comprising:

[0052] A second data acquisition unit is configured to acquire second training data, wherein the second training data includes more than one set of multi-round dialogue sample pairs and pattern information of an associated table, wherein the multi-round dialogue sample pairs include multiple sample pairs consisting of natural language texts and their corresponding structured query language SQL statements in a dialogue;

[0053] A second model training unit is configured to train a semantic parsing model including an encoder and a decoder using the second training data; during the training process, an input sequence consisting of an input natural language text, a context of the natural language text, and the pattern information is used as an input of the encoder, and the encoder outputs a vector representation of the natural language text; the decoder outputs an SQL statement using the vector representation of the natural language text, and the training goal of the semantic parsing model is to minimize the difference between an output result of the decoder and an SQL statement corresponding to the input natural language text in the second training data;

[0054] Among them, the encoder adopts the pre-trained language model established by the device provided by the fourth aspect above.

[0055] In a sixth aspect, a semantic parsing device is provided, the device comprising:

[0056] An input acquisition unit configured to acquire a natural language text corresponding to a question input by a user, a context of the natural language text in the same conversation, and pattern information of an associated table;

[0057] A semantic parsing unit, configured to input an input sequence consisting of the natural language text, the context of the natural language text, and the pattern information into a semantic parsing model to obtain an SQL statement output by the semantic parsing model;

[0058] Among them, the semantic parsing model is pre-established by the device provided by the fifth aspect above.

[0059] According to the seventh aspect, a pre-trained language model is provided, which is pre-established using the method described in the first aspect, and the pre-trained language model is used to output a vector representation of the natural language text for an input sequence consisting of the natural language text, the context of the natural language text, and the pattern information of an associated table.

[0060] According to an eighth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the first aspect are implemented.

[0061] According to a ninth aspect, there is provided an electronic device, characterized in that it includes:

[0062] one or more processors; and

[0063] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the first aspects above.

[0064] The following advantages can be achieved according to the specific embodiments provided in this application:

[0065] 1) The present application provides a method suitable for modeling structured tabular data, and the modeling method utilizes the natural language text and its context in multiple rounds of dialogue samples and the pattern information of the associated table for multi-task training, which can learn the semantic relationship between multiple rounds of questions and tables, thereby enhancing the semantic understanding ability of the pre-trained language model.

[0066] 2) This application can automatically generate multi-round dialogue sample pairs, thereby providing training data for the training of pre-trained language models and semantic parsing models that support multi-round dialogues.

[0067] 3) This application can automatically generate a large number of sample pairs consisting of SQL statements and their natural language texts as training data for the semantic parsing model, saving costs and improving efficiency, and facilitating large-scale promotion. In addition, the accuracy of the natural language text is guaranteed by pre-setting the speech template to generate the natural language text, and the diversity of the natural language text is guaranteed by fine-grained division of the speech template.

[0068] 4) This application quickly fine-tunes the pre-trained language model to obtain a semantic parsing model, which is suitable for multi-round dialogue scenarios for table queries and can better understand user questions.

[0069] 5) This application uses historical SQL statements or semantic graphs of historical SQL statements as the context of natural language text, which can alleviate the noise impact caused by the corresponding models of editing strategies such as restart and rejection of recognition in the context of multi-round conversations, and improve modeling capabilities.

[0070] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0072] Figure 1 An exemplary system architecture diagram to which the embodiments of the present application can be applied is shown;

[0073] Figure 2 A flow chart of a method for establishing a pre-trained language model provided in an embodiment of the present application;

[0074] Figure 3A flow chart of a method for obtaining first training data provided in an embodiment of the present application;

[0075] Figure 4 A flow chart of a method for generating context-independent sample pairs provided in an embodiment of the present application;

[0076] Figure 5 Part of the schematic content of an SQL syntax tree provided in an embodiment of the present application;

[0077] Figure 6 A flow chart of a method for generating multiple rounds of sample pairs provided in an embodiment of the present application;

[0078] Figure 7 A schematic diagram of the structure of a pre-trained language model provided in an embodiment of the present application;

[0079] Figure 8 A schematic diagram of the conversation semantics parsing task provided in an embodiment of the present application;

[0080] Fig. 9 A schematic diagram of the dialogue editing strategy prediction task provided in an embodiment of the present application;

[0081] Fig.10 A schematic diagram of the mask prediction task provided in an embodiment of the present application;

[0082] Fig.11 A schematic diagram of the column label prediction task provided in an embodiment of the present application;

[0083] Fig.12 A schematic diagram of using a semantic graph of historical SQL statements as a context provided in an embodiment of the present application;

[0084] Fig.13 A flowchart of a method for suggesting a semantic parsing model provided in an embodiment of the present application;

[0085] Fig.14 A schematic diagram of the structure of the semantic parsing model provided in the embodiment of the present application;

[0086] Fig.15 A flowchart of a semantic parsing method provided in an embodiment of the present application;

[0087] Fig.16 A schematic block diagram of an apparatus for establishing a pre-trained language model according to an embodiment is shown;

[0088] Fig.17 A schematic block diagram showing an apparatus for establishing a semantic parsing model according to an embodiment;

[0089] Fig.18 A schematic block diagram of a semantic parsing device according to an embodiment is shown;

[0090] Fig.19 It is a schematic diagram of the architecture of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0091] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0092] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0093] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0094] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0095] Figure 1 An exemplary system architecture to which the embodiments of the present application can be applied is shown.

[0096] like Figure 1 As shown, the system architecture may include terminal devices 101 and 102, a network 103 and a server 104. The network 103 is used to provide a medium for communication links between the terminal devices 101, 102 and the server 104. The network 103 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0097] The user can use the terminal devices 101 and 102 to interact with the server 104 through the network 103. Various applications can be installed on the terminal devices 101 and 102, such as voice interaction applications, web browser applications, communication applications, etc.

[0098] The terminal devices 101 and 102 may be various electronic devices, which may be devices with screens or devices without screens, including but not limited to smart phones, tablet computers, smart speakers, smart TVs, PCs (Personal Computers), wearable devices, and the like.

[0099] Server 104 can be a single server, a server group consisting of multiple servers, or a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product in a cloud computing service system to solve the defects of difficult management and weak service scalability in traditional physical hosts and virtual private servers (VPs, Virtual Private Server) services. The device for establishing a pre-trained language model, the device for establishing a semantic parsing model, or the semantic parsing device provided in this application can be set and run in the above-mentioned server 104. It can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module, which is not specifically limited here.

[0100] The device for establishing a pre-trained language model in the server 104 establishes a pre-trained language model in the manner provided in the present application. The device for establishing a semantic parsing model in the server 104 performs further fine-tuning training on the basis of the established pre-trained language model, thereby obtaining a semantic parsing model. After the server 104 receives the question from the terminal device 101 or the terminal device 102, the semantic parsing device in the server 104 can use the pre-established semantic parsing model to parse the question input by the user, convert it into a corresponding SQL statement, obtain the answer corresponding to the question by searching the associated table, and return the answer to the terminal device 101 or the terminal device 102 that sent the question.

[0101] The questions and answers may be in text or voice. If voice is used, the server 104 may also include a corresponding voice processing part, such as voice analysis, voice synthesis, etc., which is not limited in this application.

[0102] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0103] Figure 2 A flow chart of a method for establishing a pre-trained language model provided in an embodiment of the present application. The method can be performed by Figure 1 The server 104 in the illustrated architecture is executed. In addition to the server, other computer devices with strong computing capabilities can also be used to execute. Figure 2As shown in , the method may include the following main steps:

[0104] Step 201: Acquire first training data, where the first training data includes more than one set of multi-round dialogue sample pairs and pattern information of associated tables, where the multi-round dialogue sample pairs include multiple sample pairs consisting of natural language texts and their corresponding structured query language SQL statements in a dialogue.

[0105] Step 202: A pre-trained language model including an embedding layer and a transformer network is obtained by training with the first training data, and the pre-trained language model is used to extract a vector representation of an input natural language text; during the training process, an input sequence consisting of the natural language text, the context of the natural language text, and the pattern information of the associated table is used as an input of the pre-trained language model to perform at least one of a conversation semantic parsing task, a dialogue editing strategy prediction task, a mask prediction task, and a column label prediction task.

[0106] It can be seen that the present application provides a method suitable for modeling structured tabular data, and the modeling method utilizes the natural language text and its context in multiple rounds of dialogue samples and the pattern information of the associated table for multi-task training, which can learn the semantic relationship between multiple rounds of questions and tables, thereby enhancing the semantic understanding ability of the pre-trained language model.

[0107] The following describes each step in the above method in detail. First, the above step 201, namely "obtaining first training data", is described in detail.

[0108] For pre-trained language models, their establishment requires a large amount of training data, namely<SQL,Text> The sample pairs in this embodiment of the present application involve multiple groups of multi-round dialogue sample pairs. For example, a group of multi-round dialogue sample pairs includes three rounds of sample pairs, namely<SQL1,Text1> ,<SQL2,Text2> and<SQL3,Text3> .

[0109] It should be noted here that the limitations such as "first" and "second" involved in the embodiments of the present application do not have limitations in terms of size, order and quantity, and are only used to distinguish them in name. For example, "first training data" and "second data" are used to distinguish the training data used to establish different models.

[0110] If the first training data is obtained by manual annotation, there will be problems of high cost and low efficiency, and it cannot be promoted on a large scale. Therefore, the embodiment of the present application provides a preferred implementation method to automatically generate a large number of multi-round dialogue sample pairs as the first training data. Figure 3A flow chart of a method for obtaining first training data provided in an embodiment of the present application is as follows: Figure 3 As shown in , the method may include the following steps:

[0111] Step 301: Acquire multiple sample pairs consisting of natural language text and SQL statements.

[0112] Step 302: taking each sample pair as a sample pair for the first round of dialogue, and generating sample pairs for subsequent N rounds of dialogue, where N is a positive integer;

[0113] Step 303: The sample pairs of the first round of dialogue and the sample pairs of the subsequent N rounds of dialogue constitute a set of multi-round dialogue sample pairs.

[0114] The sample pairs obtained in step 301 are context-independent. Sample pairs consisting of natural language text and SQL statements can be obtained based on previous query logs or manual annotation, and then sample pairs for subsequent N rounds of dialogue are generated based on each sample pair. However, in order to improve efficiency, step 301 can be implemented as follows: Figure 4 The method shown in automatically generates a large number of context-independent sample pairs. It specifically includes the following steps:

[0115] Step 401: Obtain an SQL syntax tree, wherein the SQL syntax tree is pre-generated using SQL syntax rules.

[0116] SQL is a standard computer language for accessing and processing databases. It is used to access and operate database systems. SQL is used to execute queries against databases. It can retrieve data from databases, insert new records into databases, update data in databases, delete records from databases, create new databases, create new tables in databases, and so on. Most operations performed on databases can be completed by SQL statements.

[0117] SQL has specific grammar rules, according to which an abstract syntax tree can be generated, which contains the possible grammatical structure of SQL statements. The SQL syntax tree mainly includes operation nodes and condition nodes.

[0118] Among them, the operation part nodes mainly include operation keywords, focus parameters and aggregate functions. Operation keywords are keywords used to indicate specific operations, such as "select", "from", "update", "delete", etc. (SQL syntax is not case-sensitive). Aggregate functions can be some specific function names, such as avg, max, min, etc. The focus parameter mainly reflects the focus of the SQL query. In the SQL syntax tree, the focus parameter node can indicate the corresponding table data type, such as table name, column name, etc.

[0119] The conditional part nodes mainly include conditional keywords, conditional parameters and conditional operators. Conditional keywords are keywords used to indicate conditions, such as "where". Conditional operators can be some specific function names, logical operators or mathematical operators, such as and, or, not, >, <, =, etc.

[0120] Figure 5 This is a partial schematic content of a SQL syntax tree provided in an embodiment of the present application, such as Figure 5 As shown in . For the select statement, it may include an operation part node and a condition part node. Among them, the operation part node includes the operation keywords "select" and "from". The fields part after select may include the focus parameter "fieldname", which may contain indication information and may indicate the corresponding column name. The fields part may also include the function name function(), which may specifically include functionA(), functionB(), etc. The tables part after from may include the focus parameter "tablename", which may contain indication information and indicate the corresponding table name. The tables part may also include other select statements. The condition part node includes the condition keyword "where". The condition part after where may include specific condition parameters, such as expA, expB, expC, etc., which may contain indication information and indicate the corresponding column name. The Condition part may also include operators such as and, or, not, >, <, =, etc.

[0121] The SQL syntax tree may also indicate that some parts are not necessary, such as the condition part, some keywords in the operation part, etc. It should be noted that Figure 5 This is only for illustration, and the specific SQL syntax tree may adopt other structures, but the spirit and principles are similar.

[0122] Step 402: Generate one or more SQL statements by combining the SQL syntax tree with the sampled table data.

[0123] In this step, various SQL statements can be obtained by traversing the SQL syntax tree, and these expressions are context-independent expressions. In the process of traversing the SQL syntax tree, for the traversed focus parameters and / or conditional parameters, table data can be sampled from the database according to the table data type indicated by the syntax tree, and the traversed focus parameters and / or conditional parameters can be filled with the sampled table data to obtain SQL statements. After the above traversal and a combination of multiple sampling, a large number of SQL statements can be obtained.

[0124] For example, in the process of traversing the SQL syntax tree, one of the SQL statements is expressed as: "select (column name) from (table name) where (column name) > (record value)". The focus parameter and condition parameter are in brackets, and are represented by the table data type indicated by them in this expression. By sampling the column name, table name, record value, etc. from the table data and filling the above expression, the following SQL statement can be obtained:

[0125] Select name from student registration table where height>180

[0126] Select name from student registration table where weight>100

[0127] Select product name for product table where price>200

[0128] …

[0129] In addition to the above-mentioned method of traversing the SQL syntax tree, all templates can also be pre-sorted according to the SQL syntax tree, and then the templates can be filled with sampled table data, or other feasible methods can be adopted, which are not listed here one by one. However, since the method of traversing the SQL syntax tree combined with sampling table data is more efficient, this method is preferred.

[0130] Step 403: Based on the pre-configured speech template, a corresponding natural language text is generated for each SQL statement to obtain a plurality of sample pairs, where the sample pairs are composed of a SQL statement and its corresponding natural language text.

[0131] After obtaining a large number of SQL statements, the semantic parsing model requires a large amount of training data.<SQL,Text> Sample pairs, so it is necessary to generate corresponding natural language texts for the large number of SQL statements that have been obtained. The so-called natural language refers to the language that evolves naturally with culture, that is, the language form commonly used by humans to express, including Chinese, English, Japanese, etc. In the embodiment of this application, Chinese is used as an example for description.

[0132] Generally, the ability of a semantic parsing model is largely determined by the accuracy and diversity of the training data. Accuracy means whether the service can be effectively supported, and diversity means whether the effect of the model can be guaranteed. In the embodiment of the present application, the accuracy of the natural language text is ensured by presetting the speech template, and the diversity of the natural language text is ensured by fine-grained division of the speech template.

[0133] First, the service provider is allowed to pre-set some speech templates to accurately describe the user's needs, which correspond to the processing of the table in the database. Specifically, in order to ensure diversity, the above speech templates can be set at three levels: word granularity, phrase granularity and sentence granularity.

[0134] First, at the word granularity, we can determine the word granularity speech templates corresponding to the focus parameters, conditional parameters, and operators (including aggregate functions and conditional operators) in the SQL statement. For example, in financial products, the word granularity speech template for the operator max can be pre-defined as "highest". However, a large number of synonyms can be accumulated in Chinese expressions, and the diversity of word granularity can be achieved by replacing synonyms. For example, after the operator max is replaced with synonyms at the word granularity, the word granularity speech template "maximum" can also be obtained.

[0135] Then, at the phrase granularity, according to the logical relationship between the focus parameter, condition parameter and operator (including aggregate function and condition operator) in the SQL statement, the determined word granularity speech template is combined to obtain more than one phrase granularity speech template. For example, in financial products, the combination of the condition parameter "yield" and the operator max can obtain the phrase granularity speech template "maximum yield". Similarly, by using the synonym replacement method, the phrase granularity speech templates "highest yield" and "most aggressive" can also be obtained.

[0136] Then, according to the logical relationship between the phrases in the SQL statement, the obtained phrase-granularity speech templates are combined to obtain more than one sentence-granularity speech template. Taking the SQL statement "select financial products where income type = principal protection and max rate of return" as an example, the phrase-granularity speech template corresponding to "income type = principal protection" is "principal protection type", the phrase-granularity speech template corresponding to "max (rate of return)" is "highest rate of return", "most aggressive", "maximum rate of return", and the phrase-granularity speech template corresponding to "select financial products" is "what financial products are there", "what financial products are there". Then after the combination, the natural language text corresponding to the SQL statement can be obtained as follows:

[0137] "What are the principal-guaranteed financial products with the highest yield?"

[0138] "What financial products are principal-guaranteed and have the highest rate of return?"

[0139] "What are the most aggressive principal-guaranteed financial products?"

[0140] "What are the principal-guaranteed financial products with the highest rate of return?"

[0141] …….

[0142] It can be seen that by combining the above-mentioned granular speech templates, data can be produced exponentially, which improves the diversity of the generated natural language text, thereby improving the model effect of the semantic parsing model trained using this as training data.

[0143] Then, the natural language text corresponding to the SQL statement can be determined from the sentence-granularity speech template. For example, all the determined sentence-granularity speech templates can be used as the natural language text corresponding to the SQL. For another example, each natural language text can be scored in combination with other natural language evaluation methods, such as scoring based on fluency, grammatical compliance, etc., and finally the sentence-granularity speech template whose score meets the preset requirements is selected as the natural language text corresponding to the SQL. The specific scoring method can adopt some existing natural language models, which will not be described in detail here.

[0144] At this point, we can get a large number of sample pairs consisting of SQL and its corresponding natural language text. These sample pairs are context-independent, that is, they are not generated under the premise of specific context semantics. However, in actual scenarios, users' questions are often multi-round, and usually have a correlation in context semantics, so further execution is required. Figure 3 Steps 302 and 303 shown in FIG. 3 are used to generate multi-round dialogue sample pairs.

[0145] After obtaining a large number of context-independent<SQL,Text> In addition to the sample pairs, multiple rounds of dialogue sample pairs can be generated for all sample pairs, or multiple rounds of sample pairs can be generated for only some of the sample pairs. If the latter is adopted, the context-independent<SQL,Text> At least one sample pair is sampled from the sample pairs, and a multi-round dialogue sample pair is generated for each sample pair. That is, step 302, i.e., "each sample pair is used as a sample pair for the first round of dialogue to generate a sample pair for the subsequent N rounds of dialogue" and step 303, "the sample pair for the first round of dialogue and the sample pairs for the subsequent N rounds of dialogue constitute a group of multi-round dialogue sample pairs". Where N is a positive integer.

[0146] Taking the value of N as 2 as an example, suppose one of the context-independent sample pairs is<SQL1,Text1> , then we can target<SQL1,Text1> Generate sample pairs for the second round<SQL2,Text2> Then, the third round of sample pairs is generated.<SQL3,Text3> The generation process is described in detail below.

[0147] like Figure 6 As shown in , the above step 302 may specifically include the following steps:

[0148] Step 601: Use the sample pairs of the first round of dialogue as the sample pairs of the current round.

[0149] For example,<SQL1,Text1> First, it is used as the sample pair of the current round.

[0150] Step 602: Determine an editing strategy applicable to the sample pairs of the current round from a preset editing strategy set.

[0151] After analyzing multiple rounds of questions that occurred in actual scenarios, it was found that most of the questions were obtained by making some edits (e.g., deleting some conditions, modifying some conditions, etc.) based on the questions in the previous round. Therefore, in the embodiment of the present application, the relationship between each round in the actual scenario can be analyzed in advance, and some editing strategies can be pre-set to form an editing strategy set. These editing strategies in the editing strategy set reflect the semantic association between the questions of adjacent rounds, that is, what type of editing has been performed semantically.

[0152] The above editing strategies may include but are not limited to: adding conditions, modifying conditions, deleting conditions, modifying focus, modifying aggregate functions, deleting focus, restarting, rejecting recognition, and switching tables, which will be described in detail in the next step.

[0153] Because different editing strategies can be applied to different sample pairs. For example, only SQL statements with conditions can be applied to editing strategies for modifying and deleting conditions. Only SQL statements with aggregate functions can be applied to statements for modifying aggregate functions. Therefore, the corresponding applicable editing strategies can be set in advance for the syntax of various SQL statements (the grammatical structure of SQL statements and their corresponding natural language texts is similar, and can also be set for the grammar of natural language texts). In this step, the editing strategy applicable to the current round of sample pairs is first determined.

[0154] Step 603: Sample an editing strategy from the determined editing strategies, and use the sampled editing strategy to edit the SQL statements and natural language texts in the sample pairs of the current round, respectively, to obtain the edited SQL statements and natural language texts as sample pairs for the next round of dialogue.

[0155] Assume that<SQL1,Text1> The applicable editing strategies include: editing strategy 1, editing strategy 2, and editing strategy 3. Sample an editing strategy from them, for example, randomly sample editing strategy 1, and use editing strategy 1 to edit SQL1 and Text1 respectively to obtain SQL2 and Text2, and then<SQL2,Text2> It serves as the second round of sample pairs.

[0156] The following are examples of each editing strategy:

[0157] Example 1: SQL1 is "select financial products where yield>5%", and Text1 is "What financial products have a yield greater than 5%". If the sampling editing strategy is to add conditions, SQL2 can be obtained as "select financial products where yield>5%and yield type=guaranteed principal", and Text2 is "What financial products have a yield greater than 5% and are guaranteed principal". Among them, the conditions added in the SQL statement must be consistent with the conditions added in the natural language text. The condition can be randomly selected from the SQL syntax tree and filled in after sampling the column names and records in the table data, which is similar to the method in the previous SQL generation process. The phrase-granular speech template corresponding to the added condition is supplemented in the natural language text to obtain the edited natural language text.

[0158] Example 2: SQL1 is "select financial products where yield>5%", and Text1 is "What financial products have a yield greater than 5%?" If the sampling editing strategy is to modify the condition, the new column name and record value can be sampled to modify the condition, and SQL2 is obtained as "select financial products where withdrawal rate>20%." In the natural language text, the phrase granularity speech template corresponding to the original condition is replaced with the phrase granularity speech template of the modified condition, and Text2 is "What financial products have a withdrawal rate greater than 20%?"

[0159] Example 3: SQL1 is "select financial products where yield>5%", and Text1 is "What financial products have yields greater than 5%?" If the sampling editing strategy is the deletion condition, SQL2 can be obtained as "select financial products" and Text2 as "What financial products have".

[0160] Example 4: SQL1 is "select financial products where yield>5%", and Text1 is "What financial products have a yield greater than 5%?" If the sampling editing strategy is to modify the focus, the focus parameters can be modified using the new table name and column name. For example, SQL2 can be generated as "select stocks where yield>5%." The natural resources text is replaced with the phrase granularity speech template corresponding to the new focus parameter, and Text2 is obtained as "What stocks have a yield greater than 5%?"

[0161] Example 5: SQL1 is "select count financial products where yield>5%", and Text1 is "how many financial products have a yield greater than 5%". If the sampling editing strategy is to modify the aggregation function, the new aggregation function can be used to replace the aggregation function in SQL1, and SQL2 is obtained as "select top5 financial products where yield>5%. The natural language text is replaced with the word granularity speech template corresponding to the new aggregation function, and Text2 is obtained as "what are the top 5 financial products with a yield greater than 5%".

[0162] Example 6: SQL1 is "select financial products count financial products where yield>5%", and Text1 is "What financial products have a yield greater than 5% and how many are there". If the sampling editing strategy is to delete the focus, at least one focus can be deleted from SQL1, but the new SQL2 must still have the focus. For example, SQL2 is "select financial products where yield>5%", and Text2 is "What financial products have a yield greater than 5%".

[0163] Example 7: SQL1 is "select financial products where yield > 5%", Text1 is "What financial products have a yield greater than 5%?" If the sampling editing strategy is restart, then a new SQL2 is generated, which is "select count stocks where issuance time > 3 years", and Text2 is "How many stocks have been issued for more than 3 years?" Restarting is equivalent to generating a new SQL statement and natural language text, which can be reused in this case. Figure 2 Steps 201 to 202 shown in the first step generate SQL statements, and then reuse Figure 2 Step 203 shown in the figure generates a corresponding natural language text for the newly generated SQL statement. However, it should be noted that the query table is not changed in this case.

[0164] Example 8: SQL1 is "select financial products where yield>5%", and Text1 is "what are the financial products with yield greater than 5%?" If the sampling editing strategy is to reject recognition, SQL2 and Text2 will not be generated, that is,<SQL1,Text1> The next round is empty, or the next round is still<SQL1,Text1> .

[0165] Example 9: SQL1 is "select financial products where yield>5%", Text1 is "What financial products have a yield greater than 5%?" If the sampled editing strategy is table switching, a new SQL2 is generated as "select song name where singer = XX and lyrics = YY", Text2 is "What is the song sung by XX and composed by YY?" This situation is similar to the restart in the above example 7, except that the query table needs to be changed in this case.

[0166] It should be noted that the above editing strategies are merely examples listed in this embodiment, and the above editing strategies can also be expanded. As long as they are within the spirit and principles of this application, they are all within the protection scope of this application.

[0167] Step 604: Determine whether sample pairs of N rounds after the first round of dialogue have been obtained. If not, execute step 605; if yes, execute step 606.

[0168] Step 605: Use the edited sample pairs of the next round of dialogue as the sample pairs of the current round, and go to step 402 for execution.

[0169] Step 606: The sample pairs of the first round of dialogue and the sample pairs of the subsequent N rounds of dialogue constitute a set of multi-round dialogue sample pairs, and the process ends.

[0170] Assume N is 2,<SQL1,Text1> As the current round, SQL1 and Text1 are edited based on the sampling editing strategy, and the obtained<SQL2,Text2> . Then in<SQL2,Text2> As the current round, SQL2 and Text2 are edited based on the sampling editing strategy, and the obtained<SQL3,Text3> . This gives us a set of multi-round sample pairs:

[0171] <SQL1,Text1> ,<SQL2,Text2> and<SQL3,Text3> .

[0172] The above is the process of obtaining the first training data. It should be noted that, in addition to the above-mentioned method of automatically generating multi-round dialogue sample pairs, some special or typical multi-round dialogue sample pairs can also be manually labeled to be added to the first training data, so as to make the first training data richer and able to cover some special or typical situations.

[0173] The following is a description of the embodiments of the present invention. Figure 2 Step 202, namely "using the first training data to train to obtain a pre-trained language model including an Embedding layer and a Transformer network", is described in detail.

[0174] First, the structure of the pre-trained language model involved in this application is briefly described. Figure 7 As shown in , the pre-trained language model involved in the embodiments of the present application mainly includes an Embedding layer and a Transformer network. The input of the pre-trained language model is a sequence consisting of a natural language text, the context of the natural language text, and the pattern information of the associated table, such as a sequence obtained by splicing the above text and information in sequence, which is referred to as an input sequence in subsequent embodiments. The output is a vector representation of the natural language text. The pre-trained language model is a model obtained by pre-training using large-scale training data to provide a basis for fine-tuning training of downstream application models and to speed up the training of downstream application models.

[0175] In the above-mentioned pre-trained language model, the Embedding layer and the Transformer network are both commonly used networks. The Embedding layer is used to embed each character of the input sequence to obtain the Embedding result of each character. Among them, the Embedding process may include word Embedding, position Embedding and sentence Embedding. Among them, word Embedding is to encode each character or separator into a word vector to obtain a word vector representation. Position Embedding is to encode the position of each character or separator in the input sequence to obtain a representation of the position. Sentence Embedding is to encode the sentence to which each character or separator belongs to obtain the encoded representation of the sentence (i.e., string). The Transformer network is an encoding network that combines the attention mechanism and the feedforward neural network. The Transformer network is used to encode based on the Embedding result of each character, and the encoded result is used as a vector representation of the natural language text. Based on this structure, this application proposes a new training method, during which at least one of the conversational semantic parsing task, the dialogue editing policy prediction task, the mask language modeling task and the column contextual prediction task is performed.

[0176] In an embodiment of the present application, the context of a natural language text may be the historical natural language text of the natural language text in a conversation, or the historical SQL statements of the natural language text in a conversation, or the semantic graph corresponding to the historical SQL statements of the natural language text in a conversation.

[0177] Let's first take the historical natural language text of the natural language text in a conversation as an example. Assume that a natural language text in a set of multi-round conversation sample pairs sampled from the first training data is: "What is the lowest threshold?" It is difficult to analyze the user's intention simply from such a question, so it is necessary to combine the historical natural language text of the natural language text in the same conversation, for example, "What are the financial products with returns greater than 5 points?" There may be more rounds of natural language text before this, and here only the natural language text with one historical round before is taken as an example.

[0178] The association table of the natural language text is to specifically query which table in the database can answer the question corresponding to the natural language text. This can be pre-configured or determined by keyword matching, etc., and the present application does not limit the method of determining the association table. The pattern information of the association table mainly reflects the composition of the association table, which can usually be reflected by the table name, column name, etc. As one of the feasible methods, the column names of the association table can be spliced ​​into a sequence, and the sequence is spliced ​​with the sequence corresponding to the natural language text and its context to form an input sequence. For example, after splicing, the input sequence "<\s>The lowest threshold<\s>What are the financial products with returns greater than 5 points<\s>Product name<\s>Rate<\s>Risk level<\s>Starting amount<\s>…<\s>" can be obtained, where <\s> is a separator between character strings.

[0179] As a preferred implementation, the above four training tasks are used for joint training during the training process to obtain a pre-trained language model. The above four training tasks are described in detail below.

[0180] Conversation semantic parsing task:

[0181] This task is mainly used to predict the semantic relationship between the input natural language text and its context and pattern information. The semantic relationship is mainly reflected in the relationship between the words in the natural language text and its context and the column names of the associated table.

[0182] Take the natural language text "What are the financial products with returns greater than 5 points" as an example. The relationship between "5 points" and "return rate" is a semantic relationship between conditions and values, which can be represented by the label "where-value". In "What is the lowest threshold?", the relationship between "threshold" and "starting amount" is a semantic relationship between operations and columns, which can be represented by the label "select-column", and the relationship between "lowest" and "starting amount" is a semantic relationship between operations and aggregation functions, which can be represented by the label "select-agg".

[0183] The conversation semantic parsing task can be understood as adding a multi-classification network after the pre-trained language model (in order to distinguish it from the multi-classification network used in subsequent tasks, it is called the "first multi-classification network" here) to predict the above-mentioned labels, that is, to classify the relationship between each word and each column, and map the classification results to each semantic label. If no label is obtained, it can be marked as none. As a preferred embodiment, the first multi-classification network used can be a dual affine neural network. Accordingly, in the above-mentioned first training data, it is necessary to pre-annotate the semantic relationship between the natural language text and its context and the associated table model information. The training goal of the conversation semantic parsing task is to minimize the difference between the predicted results and the corresponding annotations in the first training data.

[0184] like Figure 8 As shown in , the input sequence passes through the Embedding layer and the Transformer network to output the encoding result, and the encoding result passes through the first multi-classification network to output the relationship label between the natural language text and the words in its context and the column names of the associated table. The loss function loss1 can be designed in advance using the training goal of the conversation semantic parsing task. The training iteration process updates the model parameters, including the model parameters of the Embedding layer, the Transformer network, and the first multi-classification network, using methods such as gradient descent according to the value of loss1.

[0185] As one of the feasible ways, the above-mentioned semantic relationship can be represented in the form of a semantic dependency graph. The semantic dependency graph includes nodes and edges, the nodes include words in the natural language text and its context, and the column names of the association table, and the edges indicate the semantic relationship between the nodes. Accordingly, the first training data includes a semantic dependency graph annotated with the pattern information of the natural language text and the association table in each sample pair. The conversation semantic parsing task is used to predict the semantic dependency graph of the input natural language text and its context and the pattern information of the association table, and the training goal is to minimize the difference between the predicted result and the corresponding annotation in the training sample.

[0186] Dialogue editing strategy prediction task:

[0187] This task is used to predict the editing strategy adopted by the input natural language text compared to its context. As mentioned in the previous description of obtaining the first training data, in multiple rounds of dialogue, a natural language text has a specific editing strategy relative to the natural language text of the previous round, and the editing strategy belongs to one of the preset editing strategy sets. The editing strategy of a natural language text relative to the natural language text of the previous round can help to better understand the semantics of the current natural language text. Therefore, this task is actually to better understand the semantics of the natural language text by learning the editing strategy between the two rounds. In the first training data, the editing strategy of the natural language text compared to its context (usually referring to the natural language text of the previous round) is known and can be marked in advance.

[0188] The dialog editing strategy prediction task can be understood as adding a second multi-classification network after the pre-trained language model to predict the above editing strategy, that is, classifying the editing strategy adopted by the natural language text compared with its context, and mapping the classification results to specific editing strategies. The training goal of the conversation semantic parsing task is to minimize the difference between the prediction result and the corresponding annotation (annotated editing strategy label) in the first training data.

[0189] like Fig. 9 As shown in , the input sequence passes through the Embedding layer and the Transformer network to output the encoding result, and the encoding result passes through the second multi-classification network to output the editing strategy of the natural language text compared with its context. The loss function loss2 can be designed in advance using the training goal of the dialogue editing strategy prediction task. The training iteration process updates the model parameters, including the model parameters of the Embedding layer, the Transformer network, and the second multi-classification network, using methods such as gradient descent according to the value of loss2.

[0190] Mask prediction task:

[0191] The mask prediction task is used to predict the content of the masked characters in the input sequence. The mask prediction task is to predict the content of the masked characters after randomly masking the characters in the input sequence. The semantics of the natural language text is understood by learning the content of the masked characters.

[0192] The mask prediction task can be understood as adding a third multi-classification network after the pre-trained language model to perform the above-mentioned masked character prediction, that is, classifying the masked content and mapping the classification results to a specific character space, thereby predicting a specific character. The training goal of the mask prediction task is to minimize the difference between the prediction result and the actual masked character.

[0193] like Fig.10As shown in , the input sequence outputs the encoding result after passing through the Embedding layer and the Transformer network, and the encoding result outputs the masked character content after passing through the third multi-classification network. Fig.10 In the example shown in , [mask] in the input sequence represents the masked character position, and its corresponding target prediction result should be "financial products". The loss function loss3 can be designed in advance using the training objective of the mask prediction task. The training iteration process updates the model parameters according to the value of loss3 using methods such as gradient descent, including the model parameters of the Embedding layer, the Transformer network, and the third multi-classification network.

[0194] Column label prediction task:

[0195] The column label prediction task is used to predict the column names of the associated table covered by the input natural language text. Still taking the input sequence in the above example as an example, in multiple rounds of dialogue, the column names of the associated table covered by the SQL statement corresponding to "what is the lowest threshold" include product name, rate of return, and starting price. In the first training data, the column names of the associated table covered by the SQL statement corresponding to each natural language text can be labeled.

[0196] The column label prediction task can be understood as adding a fourth multi-classification network after the pre-trained language model to predict the above column labels, that is, to classify whether the input natural language text covers each column label. The training goal of the column label prediction task is to minimize the difference between the prediction result and the corresponding annotation (annotated column label) in the first training data.

[0197] like Fig.11 As shown in , the input sequence outputs the encoding result after passing through the Embedding layer and the Transformer network, and the encoding result outputs the column name of the associated table covered by the SQL statement corresponding to the natural language text after passing through the fourth multi-classification network. The loss function loss4 can be designed in advance using the training objective of the column label prediction task. The training iteration process updates the model parameters, including the model parameters of the Embedding layer, the Transformer network, and the fourth multi-classification network, using methods such as gradient descent according to the value of loss4.

[0198] When the above four training tasks are combined at the same time, the total loss function Loss can be determined based on the loss function of each task, for example:

[0199] Loss=a1*loss1+a2*loss2+a3*loss3+a4*loss4

[0200] Among them, the above a1, a2, a3 and a4 are preset weighting coefficients.

[0201] In each round of iteration, the total loss function value is used to update the model parameters by methods such as gradient descent until the preset training end conditions are met. The training end conditions may include, for example, the total loss function value is less than or equal to the preset loss function threshold, the number of iterations reaches the preset number threshold, etc.

[0202] The above examples all take the context of natural language text as the historical natural language text in the same conversation. In some conversations, there may be some contexts that are semantically unrelated, and the corresponding editing strategies are such as restarting and rejecting recognition. If natural language text and its historical natural language text are directly learned, noise may be introduced. In view of this, the context of natural language text can use SQL information. The historical SQL statements of natural language text in the same conversation can be used, or the semantic graph corresponding to the historical SQL statements can be used.

[0203] The semantic graph of the SQL statement may include nodes and edges. The nodes include operation keywords, condition keywords, focus parameters and condition parameters; the edges may include aggregation functions and condition operators.

[0204] When historical SQL statements are used to replace historical natural language texts, the training process and processing methods of the four training tasks are basically the same and will not be described in detail here.

[0205] When the semantic graph of historical SQL statements is used to replace the historical natural language text, what is actually spliced ​​into the input sequence is the semantic graph information of the historical SQL statements in the natural language text. The Embedding layer and the Transformer network embed and encode each node and edge in the semantic graph. Since the semantic graph of the historical SQL statements already contains the relationship with the pattern information of the associated table, in the conversation semantic parsing task, it is actually only necessary to predict the semantic relationship between the input natural language text and the pattern information, such as Fig.12 The training process and processing methods of the other three training tasks are basically the same and will not be described here.

[0206] Using historical SQL statements or semantic graphs of historical SQL statements as the context of natural language text can alleviate the noise impact of the corresponding models of editing strategies such as restart and rejection of recognition in the context of multi-round conversations, and improve modeling capabilities.

[0207] The pre-trained language model established above can output a vector representation of a natural language text based on an input sequence consisting of a natural language text, the context of the natural language text, and the pattern information of the associated table. After the pre-trained language model is obtained through training through the process shown in the above embodiment, fine-tuning training can be performed based on the pre-trained language model in a specific service scenario to establish a semantic parsing model. The process is described in detail below in conjunction with a specific embodiment. Fig.13 A flowchart of a method for suggesting a semantic parsing model provided in an embodiment of the present application, which can be Figure 1 Server-side execution in the architecture shown. Fig.13 As shown in , the method may include the following steps:

[0208] Step 1301: Obtain second training data, where the second training data includes more than one set of multi-round dialogue sample pairs and pattern information of associated tables, where the multi-round dialogue sample pairs include multiple sample pairs consisting of natural language texts and their corresponding structured query language SQL statements in a dialogue.

[0209] The semantic parsing model trained in this embodiment needs to be applied to specific services, and a high model effect needs to be guaranteed. Therefore, this embodiment actually trains the semantic parsing model more accurately, and accordingly, the second training data used needs to be more accurate. In addition, since the fine-tuning training process is carried out on the basis of the already trained pre-trained language model, the requirement for the amount of training data is relatively low, and fine-tuning training can be achieved with small sample data. Based on the above two considerations, the second training data can be obtained by manual participation.

[0210] As one of the feasible methods, one or more multi-round conversation sample pairs can be used from a large number of multi-round conversation sample pairs generated by the first training data. The natural language text in the sampled multi-round conversation sample pairs is manually proofread; and the proofread multi-round conversation sample pairs are used as the second training data. When sampling from the first training data, targeted selection can be made according to the specific service scenario of the semantic parsing model.

[0211] In addition to the above implementation methods, other implementation methods may also be used, such as manually constructing multi-round dialogue sample pairs, etc., which are not listed here one by one.

[0212] Step 1302: Use the second training data to train a semantic parsing model including an encoder and a decoder; during the training process, use the input natural language text, the context of the natural language text, and an input sequence consisting of pattern information as the input of the encoder, and the encoder outputs a vector representation of the natural language text; the decoder outputs an SQL statement using the vector representation of the natural language text, and the training goal of the semantic parsing model is to minimize the difference between the output result of the decoder and the SQL statement corresponding to the natural language text input in the second training data.

[0213] The semantic parsing model established in this embodiment is as follows: Fig.14 As shown in , it includes an encoder and a decoder. The encoder initially adopts the pre-trained language model trained in the previous embodiment. The input of the semantic parsing model is also an input sequence consisting of natural language text, the context of the natural language text, and pattern information. The target output is the SQL statement corresponding to the natural language model in the second training data.

[0214] The semantic parsing model is essentially a Seq2Seq (sequence to sequence) model. The input sequence is passed through the encoder to obtain a vector representation of the natural language text, and the decoder converts the vector representation into an output sequence, which constitutes an SQL statement. The training goal of the semantic parsing model is to minimize the difference between the output result of the decoder and the SQL statement corresponding to the natural language text input in the second training data.

[0215] During the training process, a loss function can be constructed according to the above training objectives, and the value of the loss function is used in each round of iteration to update the model parameters (including the model parameters of the encoder and decoder) by methods such as gradient descent until the preset training end conditions are met. The training end conditions may include, for example, the value of the loss function is less than or equal to the preset loss function threshold, the number of iterations reaches the preset number threshold, etc. Since the pre-trained language model can already better understand the semantics of the natural language model in the association table, the semantic parsing model can be quickly trained to achieve the training objectives based on the pre-trained language model.

[0216] Depend on Fig.13 It can be seen from the process shown that this technical solution can quickly fine-tune the training to obtain a semantic parsing model based on the established pre-trained language model. The semantic parsing model is suitable for multi-round dialogue scenarios for table queries and can better understand user questions.

[0217] After the semantic parsing model is trained, it can be used to answer user questions, that is, applied to specific TableQA service scenarios. Fig.15 A flowchart of a semantic parsing method provided in an embodiment of the present application, which can be performed as follows: Figure 1Server-side execution in the architecture shown. Fig.15 As shown in , the method may include the following steps:

[0218] Step 1501: Obtain the natural language text corresponding to the question input by the user, the context of the natural language text in the same conversation, and the pattern information of the associated table.

[0219] Step 1502: Input an input sequence consisting of a natural language text, a context of the natural language text, and pattern information into a semantic parsing model to obtain an SQL statement output by the semantic parsing model.

[0220] The structure of the semantic parsing model can be found in Fig.14 , which will not be elaborated here. For example, the user enters the question "What is the lowest threshold" in the current round, and the historical question in this dialogue is "What are the financial products with returns greater than 4 points?" The column names of the associated table include product name, rate of return, risk level, starting price, etc. The above information is concatenated to obtain the input sequence "<\s>What is the lowest threshold<\s>What are the financial products with returns greater than 4 points<\s>Product name<\s>Rate<\s>Risk level<\s>Starting price". After being parsed by the semantic parsing model, the output SQL statement is "select product name min starting price where rate of return>4%".

[0221] Furthermore, step 1503 may be performed: querying the associated table using a SQL statement, and generating an answer corresponding to the question using the query result.

[0222] For example, the SQL statement "select product name min starting amount where rate of return > 4%" is used to query the wealth management product database table, and then the product name obtained from the query is used to generate an answer and return it to the user. The returned answer can be generated in natural language based on the pre-set words and then returned.

[0223] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0224] According to an embodiment of another aspect, a device for establishing a pre-trained language model is provided. Fig.16 A schematic block diagram of a device for establishing a pre-trained language model according to an embodiment is shown, wherein the device is arranged at Figure 1The device may be an application located on the server side, or a plug-in or software development kit (SDK) or other functional unit in an application located on the server side, or may be located in a computer terminal with strong computing capabilities, which is not particularly limited in the embodiments of the present invention. Fig.16 As shown, the device 1600 may include: a first data acquisition unit 1610 and a first model training unit 1620. The main functions of each component unit are as follows:

[0225] The first data acquisition unit 1610 is configured to acquire first training data, where the first training data includes more than one set of multi-round dialogue sample pairs and pattern information of associated tables, where the multi-round dialogue sample pairs include multiple sample pairs in a dialogue consisting of natural language texts and their corresponding structured query language SQL statements.

[0226] The first model training unit 1620 is configured to use the first training data to train a pre-trained language model including an embedding layer and a transfomer network, where the pre-trained language model is used to extract a vector representation of an input natural language text; during the training process, the input sequence consisting of the natural language text, the context of the natural language text, and the pattern information is used as the input of the pre-trained language model to perform at least one of a conversation semantic parsing task, a dialogue editing strategy prediction task, a mask prediction task, and a column label prediction task;

[0227] The conversational semantic parsing task is used to predict the semantic relationship between the input natural language text and its context and pattern information;

[0228] The dialogue editing strategy prediction task is used to predict the editing strategy adopted by the input natural language text compared with its context;

[0229] The mask prediction task is used to predict the masked character content in the input sequence;

[0230] The column label prediction task is used to predict the column labels of the associated table covered by the SQL statement corresponding to the input natural language text.

[0231] The first data acquisition unit 1610 may specifically include: a sample pair acquisition unit 1601 and a multi-round sample generation unit 1602 .

[0232] The sample pair acquisition unit 1601 is configured to acquire a plurality of sample pairs consisting of natural language texts and SQL statements;

[0233] The multi-round sample generating unit 1602 uses each sample pair as a sample pair for the first round of dialogue to generate sample pairs for subsequent N rounds of dialogue, where N is a positive integer; the sample pair for the first round of dialogue and the sample pairs for subsequent N rounds of dialogue constitute a set of multi-round dialogue sample pairs.

[0234] Among them, the sample pair acquisition unit 1601 is specifically configured to obtain an SQL syntax tree, where the SQL syntax tree is pre-generated using SQL syntax rules; use the SQL syntax tree in combination with sampled table data to generate one or more SQL statements; based on a pre-configured speech template, generate corresponding natural language text for each SQL statement to obtain multiple sample pairs.

[0235] The multi-round sample generation unit 1602 is specifically configured to use the sample pair of the first round of dialogue as the sample pair of the current round; determine the editing strategy applicable to the sample pair of the current round from a preset editing strategy set; sample an editing strategy from the determined editing strategies, and use the sampled editing strategy to edit the SQL statement and natural language text in the sample pair of the current round, respectively, to obtain the edited SQL statement and natural language text as the sample pair of the next round of dialogue; use the sample pair of the next round of dialogue obtained by editing as the sample pair of the current round, and proceed to execute the operation of determining the editing strategy applicable to the sample pair of the current round from the preset editing strategy set, until the sample pair of N rounds of dialogue after the sample pair of the first round of dialogue is obtained.

[0236] Among them, the above-mentioned editing strategy includes at least one of the following: adding conditions, modifying conditions, deleting conditions, modifying focus, modifying aggregate functions, deleting focus, restarting, refusing recognition and table switching.

[0237] As an implementable method, the first training data also includes: a semantic dependency graph annotated with pattern information of the natural language text and the associated table in each sample pair, the semantic dependency graph including nodes and edges, the nodes including words in the natural language text and column names of the associated table, and the edges indicating the semantic relationship between the nodes.

[0238] The conversational semantic parsing task is used to predict the semantic dependency graph of the input natural language text and its context and pattern information of the associated table. The training goal of this task is to minimize the difference between the predicted results and the corresponding annotations in the training samples.

[0239] As an implementable manner, the context of the natural language text may include: historical natural language texts of the natural language text in a conversation.

[0240] As another achievable manner, the context of the natural language text may include historical SQL statements, or semantic graphs corresponding to historical SQL statements;

[0241] The semantic graph includes nodes and edges. The nodes include operation keywords, condition keywords, focus parameters and condition parameters. The edges include aggregation functions and condition operators.

[0242] Fig.17 A schematic block diagram of a device for establishing a semantic parsing model according to an embodiment is shown, wherein the device is arranged at Figure 1 The device may be an application located on the server side, or a plug-in or software development kit (SDK) or other functional unit in an application located on the server side, or may be located in a computer terminal with strong computing capabilities, which is not particularly limited in the embodiments of the present invention. Fig.17 As shown, the device 1700 may include: a second data acquisition unit 1701 and a second model training unit 1702. The main functions of each component unit are as follows:

[0243] The second data acquisition unit 1701 is configured to acquire second training data, where the second training data includes more than one set of multi-round dialogue sample pairs and pattern information of an associated table, where the multi-round dialogue sample pairs include multiple sample pairs consisting of natural language texts and their corresponding structured query language SQL statements in a dialogue;

[0244] The second model training unit 1702 is configured to train a semantic parsing model including an encoder and a decoder using the second training data; during the training process, an input sequence consisting of an input natural language text, a context of the natural language text, and pattern information is used as an input of the encoder, and the encoder outputs a vector representation of the natural language text; the decoder outputs an SQL statement using the vector representation of the natural language text, and the training goal of the semantic parsing model is to minimize the difference between the output result of the decoder and the SQL statement corresponding to the input natural language text in the second training data;

[0245] The encoder uses Fig.16 The pre-trained language model established by the device.

[0246] As one of the achievable methods, the second data acquisition unit 1701 may be specifically configured to sample one or more multi-round conversation sample pairs from the first training data, obtain manual proofreading results of the natural language texts in the sampled multi-round conversation sample pairs, and use the proofread multi-round conversation sample pairs as the second training data.

[0247] Fig.18 A schematic block diagram of a semantic parsing device according to an embodiment is shown. The device is arranged at Figure 1The device may be an application located on the server side, or a plug-in or software development kit (SDK) or other functional unit in an application located on the server side, or may be located in a computer terminal with strong computing capabilities, which is not particularly limited in the embodiments of the present invention. Fig.18 As shown, the device 1800 may include: an input acquisition unit 1801 and a semantic parsing unit 1802, and may further include a query unit 1803 and an answer generation unit 1804. The main functions of each component unit are as follows:

[0248] The input acquisition unit 1801 is configured to acquire the natural language text corresponding to the question input by the user, the context of the natural language text in the same conversation, and the mode information of the associated table;

[0249] The semantic parsing unit 1802 is configured to input an input sequence consisting of a natural language text, a context of the natural language text, and pattern information into a semantic parsing model to obtain an SQL statement output by the semantic parsing model;

[0250] The semantic parsing model consists of Fig.17 The device shown is pre-built.

[0251] The query unit 1803 is configured to query the associated table using a SQL statement.

[0252] The answer generating unit 1804 is configured to generate an answer corresponding to the question using the query result.

[0253] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0254] It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

[0255] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0256] And an electronic device, comprising:

[0257] one or more processors; and

[0258] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0259] in, Fig.19 The electronic device architecture is shown as an example, which may include a processor 1910, a video display adapter 1911, a disk drive 1912, an input / output interface 1913, a network interface 1914, and a memory 1920. The processor 1910, the video display adapter 1911, the disk drive 1912, the input / output interface 1913, the network interface 1914, and the memory 1920 may be communicatively connected via a communication bus 1930.

[0260] Among them, the processor 1910 can be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solution provided in this application.

[0261] The memory 1920 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1920 can store an operating system 1921 for controlling the operation of the electronic device 1900, and a basic input and output system (BIOS) 1922 for controlling the low-level operation of the electronic device 1900. In addition, a web browser 1923, a data storage management system 1924, and a device for establishing a pre-trained language model / a device for establishing a semantic parsing model / a semantic parsing device 1925, etc. can also be stored. The above-mentioned device for establishing a pre-trained language model / a device for establishing a semantic parsing model / a semantic parsing device 1925 can be an application program that specifically implements the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided in the present application is implemented by software or firmware, the relevant program code is stored in the memory 1920 and is called and executed by the processor 1910.

[0262] The input / output interface 1913 is used to connect the input / output module to realize information input and output. The input / output module can be configured as a component in the device (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0263] The network interface 1914 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0264] The bus 1930 comprises a pathway for transmitting information between the various components of the device (eg, the processor 1910, the video display adapter 1911, the disk drive 1912, the input / output interface 1913, the network interface 1914, and the memory 1920).

[0265] It should be noted that, although the above device only shows a processor 1910, a video display adapter 1911, a disk drive 1912, an input / output interface 1913, a network interface 1914, a memory 1920, a bus 1930, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0266] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0267] The above is a detailed introduction to the method, device, equipment and computer-readable storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A method for establishing a pre-trained language model, the method comprising: Acquire first training data, where the first training data includes more than one set of multi-round dialogue sample pairs and pattern information of an associated table, where the multi-round dialogue sample pairs include multiple sample pairs consisting of natural language texts and their corresponding structured query language SQL statements in a dialogue; Using the first training data to train a pre-trained language model including an embedding layer and a Transformer network, wherein the Transformer network is an encoding network combining an attention mechanism and a feedforward neural network, and the pre-trained language model is used to extract a vector representation of an input natural language text; During the training process, the input sequence consisting of the natural language text, the context of the natural language text, and the pattern information is used as the input of the pre-trained language model to perform at least one of a conversation semantic parsing task, a dialogue editing strategy prediction task, a mask prediction task, and a column label prediction task; The conversation semantic parsing task is used to predict the semantic relationship between the input natural language text and its context and the pattern information, wherein the context of the natural language text includes: the historical natural language text of the natural language text in a conversation, or the historical SQL statements, or the semantic graph corresponding to the historical SQL statements; The dialogue editing strategy prediction task is used to predict the editing strategy adopted by the input natural language text compared with its context; The mask prediction task is used to predict the masked character content in the input sequence; The column label prediction task is used to predict the column name of the associated table covered by the SQL statement corresponding to the input natural language text.

2. The method according to claim 1, wherein: The acquiring of the first training data comprises: Obtain multiple sample pairs consisting of natural language texts and SQL statements; Each sample pair is used as a sample pair for the first round of dialogue, and sample pairs for subsequent N rounds of dialogue are generated, where N is a positive integer; The sample pairs of the first round of dialogue and the sample pairs of the subsequent N rounds of dialogue constitute a set of multi-round dialogue sample pairs.

3. The method according to claim 2, wherein: The obtaining of a plurality of sample pairs consisting of natural language texts and SQL statements includes: Obtaining an SQL syntax tree, wherein the SQL syntax tree is pre-generated using SQL syntax rules; Generate one or more SQL statements using the SQL syntax tree in combination with the sampled table data; Based on the pre-configured speech templates, corresponding natural language texts are generated for each SQL statement to obtain multiple sample pairs.

4. The method according to claim 2, wherein: The method of using each sample pair as a sample pair for a first round of dialogue to generate sample pairs for subsequent N rounds of dialogue includes: Using the sample pairs of the first round of dialogue as sample pairs of the current round; Determine an editing strategy applicable to the sample pair of the current round from a preset editing strategy set; An editing strategy is sampled from the determined editing strategies, and the SQL statements and natural language texts in the sample pairs of the current round are edited using the sampled editing strategy, so as to obtain the edited SQL statements and natural language texts as the sample pairs of the next round of dialogue; The sample pairs of the next round of dialogue obtained by editing are used as the sample pairs of the current round, and the operation of determining the editing strategy applicable to the sample pairs of the current round from the preset editing strategy set is executed until the sample pairs of the N rounds of dialogue after the sample pairs of the first round of dialogue are obtained.

5. The method according to claim 1, wherein: The first training data also includes: a semantic dependency graph annotated with respect to the natural language text in each sample pair and the pattern information of the association table, the semantic dependency graph including nodes and edges, the nodes including words in the natural language text and the column names of the association table, and the edges indicating the semantic relationship between the nodes; The conversation semantic parsing task is used to predict the semantic dependency graph of the input natural language text and its context and the pattern information of the association table. The training goal of this task is to minimize the difference between the prediction result and the corresponding annotation in the training sample.

6. The method according to claim 1 or 4, wherein: The editing strategy includes at least one of the following: Add conditions, modify conditions, delete conditions, modify focus, modify aggregate functions, delete focus, restart, refuse recognition and switch tables.

7. The method according to claim 1, wherein: The context of the natural language text includes: historical natural language texts of the natural language text in a conversation, or historical SQL statements, or semantic graphs corresponding to historical SQL statements; The semantic graph includes nodes and edges, the nodes include operation keywords, condition keywords, focus parameters and condition parameters, and the edges include aggregation functions and condition operators.

8. A method for establishing a semantic parsing model, the method comprising: Acquire second training data, where the second training data includes more than one set of multi-round dialogue sample pairs and pattern information of an associated table, where the multi-round dialogue sample pairs include multiple sample pairs consisting of natural language texts and their corresponding structured query language SQL statements in a dialogue; The second training data is used to train a semantic parsing model including an encoder and a decoder; during the training process, an input sequence consisting of an input natural language text, a context of the natural language text, and the pattern information is used as an input to the encoder, and the encoder outputs a vector representation of the natural language text; the decoder outputs an SQL statement using the vector representation of the natural language text, and the training goal of the semantic parsing model is to minimize the difference between the output result of the decoder and the SQL statement corresponding to the input natural language text in the second training data; Wherein, the encoder adopts a pre-trained language model established by the method described in any one of claims 1 to 7.

9. The method according to claim 8, wherein: The acquiring of the second training data comprises: One or more multi-round dialogue sample pairs are sampled from the first training data, and manual proofreading results of natural language texts in the sampled multi-round dialogue sample pairs are obtained; and the proofread multi-round dialogue sample pairs are used as the second training data.

10. A semantic parsing method, the method comprising: Obtaining the natural language text corresponding to the question input by the user, the context of the natural language text in the same conversation, and the mode information of the associated table; Inputting an input sequence consisting of the natural language text, the context of the natural language text, and the pattern information into a semantic parsing model to obtain an SQL statement output by the semantic parsing model; Wherein, the semantic parsing model is pre-established using the method as claimed in claim 9.

11. The method according to claim 10, further comprising: Query the associated table using the SQL statement; The query results are used to generate answers corresponding to the questions.

12. A device for establishing a pre-trained language model, the device comprising: A first data acquisition unit is configured to acquire first training data, wherein the first training data includes more than one set of multi-round dialogue sample pairs and pattern information of an associated table, wherein the multi-round dialogue sample pairs include multiple sample pairs consisting of natural language texts and their corresponding structured query language SQL statements in a dialogue; A first model training unit is configured to use the first training data to train a pre-trained language model including an embedding layer and a Transformer network, wherein the Transformer network is an encoding network combining an attention mechanism and a feedforward neural network, and the pre-trained language model is used to extract a vector representation of an input natural language text; During the training process, the input sequence consisting of the natural language text, the context of the natural language text and the pattern information is used as the input of the pre-trained language model to perform at least one of a conversation semantic parsing task, a dialogue editing strategy prediction task, a mask prediction task and a column label prediction task, wherein the context of the natural language text includes: a historical natural language text of the natural language text in a conversation, or a historical SQL statement, or a semantic graph corresponding to the historical SQL statement; The conversation semantic parsing task is used to predict the semantic relationship between the input natural language text and its context and the pattern information; The dialogue editing strategy prediction task is used to predict the editing strategy adopted by the input natural language text compared with its context; The mask prediction task is used to predict the masked character content in the input sequence; The column label prediction task is used to predict the column labels of the associated table covered by the SQL statement corresponding to the input natural language text.

13. A device for establishing a semantic parsing model, the device comprising: A second data acquisition unit is configured to acquire second training data, wherein the second training data includes more than one set of multi-round dialogue sample pairs and pattern information of an associated table, wherein the multi-round dialogue sample pairs include multiple sample pairs consisting of natural language texts and their corresponding structured query language SQL statements in a dialogue; A second model training unit is configured to train a semantic parsing model including an encoder and a decoder using the second training data; during the training process, an input sequence consisting of an input natural language text, a context of the natural language text, and the pattern information is used as an input of the encoder, and the encoder outputs a vector representation of the natural language text; the decoder outputs an SQL statement using the vector representation of the natural language text, and the training goal of the semantic parsing model is to minimize the difference between an output result of the decoder and an SQL statement corresponding to the input natural language text in the second training data; Wherein, the encoder adopts a pre-trained language model established by the device described in claim 12.

14. A semantic parsing device, the device comprising: An input acquisition unit configured to acquire a natural language text corresponding to a question input by a user, a context of the natural language text in the same conversation, and pattern information of an associated table; A semantic parsing unit, configured to input an input sequence consisting of the natural language text, the context of the natural language text, and the pattern information into a semantic parsing model to obtain an SQL statement output by the semantic parsing model; Wherein, the semantic parsing model is pre-established by the device as described in claim 13.

15. A pre-trained language model system, pre-established using the method described in any one of claims 1 to 7, wherein the pre-trained language model is used to output a vector representation of a natural language text based on an input sequence consisting of a natural language text, a context of the natural language text, and pattern information of an associated table.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

17. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 11.

Images