Model training, natural language processing method, device, equipment and storage medium

Abstract

The present disclosure relates to a model training, natural language processing method, apparatus, device and storage medium. The present disclosure divides the process of converting natural language into SQL query statements into multiple subtasks, and trains the machine learning model through the clauses obtained by the multiple subtasks, so that the machine learning model learns each subtask. Then, the task prompt words, natural language statements, and database patterns corresponding to the multiple subtasks are input into the machine learning model, so that the machine learning model outputs structured query statements corresponding to the natural language statements. Thus, according to the structured query statements and the standard structured query statements corresponding to the natural language statements, the machine learning model is retrained, so that the machine learning model learns the main task. Through this task decomposition and combined learning method, the trained machine learning model can generate accurate SQL query statements and improve the generalization ability and robustness of the machine learning model.

Description

Technical Field

[0001] The present disclosure relates to the field of information technology, and in particular to a model training, natural language processing method, apparatus, device and storage medium. Background Art

[0002] Currently, natural language can be converted into structured query language (SQL) through machine learning models.

[0003] However, current machine learning models of this type have weak generalization and robustness. Summary of the Invention

[0004] To address, or at least partially address, the above technical issues, the present disclosure provides a model training and natural language processing method, apparatus, device, and storage medium, enabling a trained machine learning model to generate accurate SQL query statements. Furthermore, by decomposing, learning, and combining tasks, the machine learning model can learn structured knowledge of SQL query statements, thereby improving the generalization and robustness of the machine learning model.

[0005] In a first aspect, an embodiment of the present disclosure provides a model training method, comprising:

[0006] Obtain natural language sentences;

[0007] Generate multiple first sequences according to the preset multiple subtasks, each first sequence including a task prompt word corresponding to a subtask, the natural language sentence, and a database model corresponding to the natural language sentence;

[0008] inputting the plurality of first sequences into a machine learning model to be trained, so that the machine learning model generates a plurality of second sequences, wherein the plurality of first sequences correspond to the plurality of second sequences in a one-to-one manner, the second sequences including one or more query keywords and query contents corresponding to the one or more query keywords, wherein the query contents are generated based on the natural language sentence and the database schema;

[0009] Training the machine learning model according to the plurality of second sequences and the standard query content corresponding to the query keyword in each second sequence;

[0010] Inputting the task prompt words corresponding to the multiple subtasks, the natural language sentences, and the database model into the machine learning model, so that the machine learning model outputs a structured query sentence corresponding to the natural language sentence;

[0011] The machine learning model is retrained according to the structured query statement and the standard structured query statement corresponding to the natural language statement.

[0012] In a second aspect, an embodiment of the present disclosure provides a natural language processing method, including:

[0013] Obtain natural language sentences;

[0014] The task prompt words corresponding to the preset multiple subtasks, the natural language sentences and the database patterns corresponding to the natural language sentences are input into the trained machine learning model, so that the machine learning model outputs the structured query sentences corresponding to the natural language sentences. The machine learning model is trained using the method described in the first aspect.

[0015] In a third aspect, an embodiment of the present disclosure provides a model training device, comprising:

[0016] An acquisition module, used to acquire natural language sentences;

[0017] A generating module, configured to generate a plurality of first sequences according to a plurality of preset subtasks, each first sequence comprising a task prompt word corresponding to a subtask, the natural language sentence, and a database model corresponding to the natural language sentence;

[0018] an input module, configured to input the plurality of first sequences into a machine learning model to be trained, so that the machine learning model generates a plurality of second sequences, wherein the plurality of first sequences correspond to the plurality of second sequences in a one-to-one manner, the second sequences comprising one or more query keywords and query content corresponding to the one or more query keywords, wherein the query content is generated based on the natural language statement and the database schema;

[0019] a training module, configured to train the machine learning model based on the plurality of second sequences and standard query content corresponding to the query keywords in each second sequence;

[0020] The input module is further configured to input the task prompt words corresponding to the plurality of subtasks, the natural language sentences, and the database model into the machine learning model, so that the machine learning model outputs a structured query sentence corresponding to the natural language sentence;

[0021] The training module is also used to retrain the machine learning model based on the structured query statement and the standard structured query statement corresponding to the natural language statement.

[0022] In a fourth aspect, an embodiment of the present disclosure provides a natural language processing apparatus, including:

[0023] An acquisition module, used to acquire natural language sentences;

[0024] An input module is used to input task prompt words corresponding to multiple preset subtasks, the natural language sentences and the database model corresponding to the natural language sentences into a trained machine learning model, so that the machine learning model outputs a structured query sentence corresponding to the natural language sentence. The machine learning model is trained using the method described in the first aspect.

[0025] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, including:

[0026] Memory;

[0027] processor; and

[0028] computer programs;

[0029] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect or the second aspect.

[0030] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method described in the first aspect or the second aspect.

[0031] The model training and natural language processing methods, apparatuses, devices, and storage media provided by the embodiments of the present disclosure divide the process of converting natural language into SQL query statements into multiple subtasks. Based on the multiple pre-set subtasks, multiple first sequences are generated. The multiple first sequences are input into the machine learning model to be trained, so that the machine learning model generates multiple second sequences. Further, the machine learning model is trained based on the multiple second sequences and the standard query content corresponding to the query keywords in each second sequence, so that the machine learning model can first learn each subtask. Then, the task prompt words corresponding to the multiple subtasks, the natural language statements, and the database schema are input into the machine learning model, so that the machine learning model outputs structured query statements corresponding to the natural language statements. Thus, based on the structured query statements and the standard structured query statements corresponding to the natural language statements, the machine learning model is retrained, so that the machine learning model can learn the main task, that is, the process of converting natural language into SQL query statements. Through this task decomposition and combined learning method, the trained machine learning model can not only accurately learn the link relationship between user questions and table column names or values, but also generate accurate SQL query statements. In addition, by decomposing, learning, and combining tasks, the machine learning model can learn the structured knowledge of SQL query statements, thereby improving the generalization ability and robustness of the machine learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0033] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0034] Figure 1 A flow chart of the model training method provided in an embodiment of the present disclosure;

[0035] Figure 2 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0036] Figure 3 A flow chart of the model training method provided in an embodiment of the present disclosure;

[0037] Figure 4 A flow chart of a model training method provided in another embodiment of the present disclosure;

[0038] Figure 5 A flow chart of a model training method provided in another embodiment of the present disclosure;

[0039] Figure 6 A schematic diagram of the structure of a model training device provided in an embodiment of the present disclosure;

[0040] Figure 7 A schematic diagram of the structure of a natural language processing device provided in an embodiment of the present disclosure;

[0041] Figure 8 A schematic diagram of the structure of an electronic device embodiment provided by the present disclosure. DETAILED DESCRIPTION

[0042] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0043] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0044] It should be noted that the natural language statements (including but not limited to user questions, user statements, etc.) and databases (including but not limited to databases pointed to by user questions, or databases determined based on user questions, etc.) involved in this application are all information and data authorized by users or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0045] In addition, the model training and natural language processing methods provided in this application involve the following terminology explanations, which are detailed as follows:

[0046] SQL: Structured Query Language, structured query statement.

[0047] Text-To-SQL: Convert natural language into SQL query statements.

[0048] Transformer: A natural language understanding model based on attention mechanism.

[0049] Encoder-Decoder: The encoder-decoder framework is a mainstream framework for natural language generation tasks. The encoder is used to understand the input sequence, and the decoder is used to generate each character in the output sequence in sequence.

[0050] TKK: Task decomposition-Knowledge Acquisition-Knowledge Composition, a three-stage Text-To-SQL method proposed in the embodiments of this application.

[0051] Fine-tune: Fine-tune the model parameters based on new data after the model has been trained.

[0052] Tables are a widely used storage method for storing and displaying structured data. Because tables offer clear structure, ease of maintenance, and high timeliness, they are used to store large amounts of high-value data in industries like finance, insurance, and healthcare, creating large-scale databases. Table semantic parsing technology quickly adapts to databases, allowing users to automatically obtain answers from the database by simply asking natural language questions, significantly improving the efficiency and user experience of interacting with the database.

[0053] Typically, machine learning models can be used to convert natural language into structured query language (SQL). However, current machine learning models of this type have limited generalization and robustness. To address this issue, the present disclosure provides a model training method, which is described below in conjunction with specific embodiments.

[0054] Figure 1 This is a flow chart of the model training method provided in the embodiment of the present disclosure. The method can be executed by a model training device, which can be implemented in software and / or hardware. The device can be configured in an electronic device, such as a server or a terminal, wherein the terminal specifically includes a mobile phone, a computer or a tablet computer. In addition, the model training method described in this embodiment can be applied to Figure 2 The application scenario shown in Figure 2 is as follows. Figure 2 As shown, the application scenario includes a terminal 21 and a server 22, wherein the terminal 21 can provide natural language sentences, such as user questions, to the server 22. The server 22 can train the machine learning model using the model training method described in this embodiment. Figure 2 This method is described in detail, such as Figure 1 As shown, the specific steps of this method are as follows:

[0055] S101. Obtain a natural language sentence.

[0056] For example Figure 2 The terminal 21 shown collects user questions, which can be questions in the form of voice or text. Since user questions usually conform to natural language logic, user questions can be regarded as natural language sentences. However, it is understandable that natural language sentences are not limited to user questions. For example, they can also be user statements. In this embodiment, taking user questions as an example, it is assumed that the user question is "What is the name of a document that has one of the three most common types and one of the three most common structures?" Further, the terminal 21 can send the user question to the server 22. If the user question is a question in the form of voice, the server 22 can use automatic speech recognition (ASR) technology to convert the voice question into a text question.

[0057] Optionally, after obtaining the natural language statement, the method further includes: obtaining a standard structured query statement corresponding to the natural language statement, wherein the standard structured query statement includes standard query contents corresponding to a plurality of query keywords.

[0058] like Figure 3 As shown, through Text-To-SQL syntax analysis, a standard structured query statement corresponding to "What is the name of the document that has one of the three most common types and one of the three most common structures" can be obtained. In addition, in some embodiments, the standard structured query statement corresponding to the user question can also be manually generated or manually annotated. Specifically, the standard structured query statement includes standard query content corresponding to multiple query keywords. For example, SELECT is a query keyword, and "document name" is the standard query content corresponding to SELECT. FROM is a query keyword, and "document collection" is the standard query content corresponding to FROM, and so on.

[0059] S102: Generate multiple first sequences according to multiple preset subtasks, each first sequence including a task prompt word corresponding to a subtask, the natural language sentence, and a database model corresponding to the natural language sentence.

[0060] In this embodiment, Figure 3As shown in ①, the Text-To-SQL task can be broken down into multiple subtasks, such as the SELECT subtask, the FROM subtask, the WHERE subtask, the GHOL subtask, and other subtasks. Each subtask is used to convert the user query into one or more clauses, and the clauses converted by different subtasks are also different. For example, the SELECT subtask is used to convert the user query into a SELECT clause, and the FROM subtask is used to convert the user query into a FROM clause. The WHERE subtask is used to convert the user query into a WHERE clause. The GHOL subtask is used to convert the user query into a GROUP_BY clause, a HAVING clause, an ORDER_BY clause, and a LIMIT clause. Other subtasks are used to convert the user query into clauses other than the above-mentioned types of clauses.

[0061] [SELECT], [FROM], [WHERE], [GROUP_BY], [HAVING], [ORDER_BY], and [LIMIT] are SQL keywords (also called keywords). Therefore, in this embodiment, [SELECT], [FROM], [WHERE], [GROUP_BY], [HAVING], [ORDER_BY], and [LIMIT] can be recorded as query keywords.

[0062] Specifically, the server 22 may assign a task prompt word to each subtask. For example, the task prompt word assigned by the server 22 to the SELECT subtask is [SELECT], the task prompt word assigned to the FROM subtask is [FROM], the task prompt word assigned to the WHERE subtask is [WHERE], the task prompt words assigned to the GHOL subtask are [GROUP_BY], [HAVING], [ORDER_BY], [LIMIT], and the task prompt word assigned to other subtasks is [SQL]. Furthermore, the server 22 generates a first sequence for each subtask. The first sequence is a combination of the task prompt word corresponding to the subtask, the user question, and the database schema corresponding to the user question (e.g. Figure 3 {S} and {C} shown).

[0063] For example Figure 3 As shown, the first sequence corresponding to the SELECT subtask is "[SELECT] What is the name of the document that has one of the three most common types and one of the three most common structures?; {S}; {C}".

[0064] The first sequence corresponding to the FROM subtask is “[FROM] What are the names of the documents that have both one of the three most common types and one of the three most common structures?; {S}; {C}”.

[0065] The first sequence corresponding to the WHERE subtask is “[WHERE] What are the names of documents that have both one of the three most common types and one of the three most common structures?; {S}; {C}”.

[0066] The first sequence corresponding to the GHOL subtask is “[GROUP_BY], [HAVING], [ORDER_BY], [LIMIT] What are the names of documents that have one of the three most common types and one of the three most common structures?; {S}; {C}”.

[0067] The first sequence corresponding to the other subtasks is “[SQL] What is the name of the document that has one of the three most common types and one of the three most common structures?; {S}; {C}”.

[0068] Specifically, this embodiment takes two factors into consideration when constructing subtasks, one of which is the number of classification examples, and the other is the dependency between different clauses. For example, according to the grammatical rules of SQL, almost every SQL query statement contains a SELECT clause and a FROM clause, while the GROUP_BY clause and the ORDER_BY clause only appear in some more complex SQL query statements. This means that the number of SELECT clauses or FROM clauses is much greater than the number of GROUP_BY clauses or ORDER_BY clauses. Therefore, this embodiment combines the GROUP_BY clause, the HAVING clause, the ORDER_BY clause, and the LIMIT clause to construct the GHOL subtask. On the one hand, this can reduce the number of tasks so that the machine learning model can focus on learning each subtask, and on the other hand, it can balance the number of training samples between different subtasks, thereby preventing the machine learning model from only learning subtasks with a larger number of samples.

[0069] Optionally, the database schema corresponding to the natural language statement includes the identifiers of one or more tables included in the database corresponding to the natural language statement, and the column names included in each of the one or more tables.

[0070] For example, server 22 may include multiple databases, or server 22 may be connected to multiple databases. In one possible scenario, when a user asks a question, the user may determine the database for which the question is being posed. In other words, the user may specify a specific database for the question. In another possible scenario, server 22 or another server may analyze the user's question to determine the database for which the question is being posed.

[0071] After determining the database corresponding to the user question, the server 22 may generate a database schema, for example, the database corresponding to the user question includes several tables, the name of each table, the number of columns in each table, the names of the columns in each table, etc. Figure 3 As shown, {S} represents the names of all tables included in the database corresponding to the user question, and {C} represents the column names included in each table in the database corresponding to the user question. It can be understood that the database corresponding to the user question may include one table (Table) or multiple tables. In addition, the name of the table can be used as the identifier of the table, but the identifier of the table is not limited to the name of the table. For example, the table number can also be used as the identifier of the table. Different tables have different corresponding identifiers.

[0072] S103. Input the multiple first sequences into the machine learning model to be trained, so that the machine learning model generates multiple second sequences, the multiple first sequences and the multiple second sequences correspond one to one, the second sequences include one or more query keywords, and query content corresponding to the one or more query keywords, respectively, the query content is generated based on the natural language statement and the database model.

[0073] For example, Figure 3 The first sequences corresponding to the subtasks shown are input into the machine learning model to be trained, which is a TKK model. This allows TKK to generate multiple second sequences, each of which corresponds one-to-one to the first sequence. That is, TKK can generate a second sequence corresponding to each first sequence. The second sequence includes one or more query keywords and query content corresponding to the one or more query keywords. The query content is generated based on the user question and the database schema corresponding to the user question. The one or more query keywords included in the second sequence are one or more query keywords corresponding to the task prompt words in the first sequence corresponding to the second sequence.

[0074] In this embodiment, different subtasks focus on different points. For example, the SELECT subtask and the WHERE subtask focus on the link between the user query and the table column names, while the [ORDER_BY] and [LIMIT] in the GHOL subtask focus on understanding the user query.

[0075] like Figure 3 As shown, for the first sequence corresponding to the SELECT subtask, the second sequence generated by TKK includes [SELECT] and the query content corresponding to [SELECT], such as "document name". The second sequence is the SELECT clause described above.

[0076] For the first sequence corresponding to the FROM subtask, the second sequence generated by TKK includes [FROM] and the query content corresponding to [FROM], such as "document collection." The second sequence is the FROM clause as described above.

[0077] Since TKK cannot determine the query content corresponding to [WHERE] based on the user question and the database schema corresponding to the user question, the second sequence generated by TKK only includes [WHERE] for the first sequence corresponding to the WHERE subtask. This second sequence is the WHERE clause described above.

[0078] In addition, since the task prompts corresponding to the GHOL subtask include multiple query keywords such as [GROUP_BY], [HAVING], [ORDER_BY], and [LIMIT], TKK needs to determine the query content corresponding to [GROUP_BY], [HAVING], [ORDER_BY], and [LIMIT] respectively based on the user question and the database schema corresponding to the user question for the first sequence corresponding to the GHOL subtask. For example, the query content corresponding to [GROUP_BY] is "document type code", there is no corresponding query content for [HAVING], and the query content corresponding to [ORDER_BY] is "count(*)desc", where count(*) represents a counting function and desc represents descending sorting. The query content corresponding to [LIMIT] is 3. Therefore, for the first sequence corresponding to the GHOL subtask, the second sequence generated by TKK includes [GROUP_BY], [HAVING], [ORDER_BY], [LIMIT], and the query contents corresponding to [GROUP_BY], [HAVING], [ORDER_BY], and [LIMIT]. Among them, "[GROUP_BY] document type code" is the GROUP_BY clause described above, "[HAVING]" is the HAVING clause described above, "[ORDER_BY] count(*)desc" is the ORDER_BY clause described above, and "[LIMIT] 3" is the LIMIT clause described above.

[0079] In addition, according to the user's question "What is the name of the document that has one of the three most common types and one of the three most common structures?", it can be seen that the document finally found needs to meet two conditions at the same time, one condition is that it has one of the three most common types, and the other condition is that it has one of the three most common structures. If one condition can generate an SQL query statement, then the user's question needs to correspond to two SQL query statements. In order to combine these two SQL query statements, this embodiment uses the collective operator [INTERSECT] to connect the two SQL query statements. Among them, the first of the two SQL query statements is regarded as a regular query statement, and the second SQL query statement can be used as an SQL clause of the first SQL query statement. Therefore, if Figure 3 The other subtasks shown can be used to convert the user question into an SQL clause, such as Figure 3 The content in the box 31 shown. That is to say, for the first sequence corresponding to other subtasks, the second sequence generated by TKK is the SQL clause. That is to say, although the task prompt word corresponding to other subtasks is [SQL], since other subtasks can convert the user question into an SQL clause, and the SQL clause is essentially an SQL query statement, and the SQL query statement itself can include multiple query keywords, therefore, in addition to the connective query keyword [INTERSECT], the SQL clause can also include other query keywords and query content corresponding to other query keywords. Therefore, the query keyword corresponding to the task prompt word [SQL] includes not only [INTERSECT], but also other query keywords.

[0080] S104: Train the machine learning model according to the multiple second sequences and the standard query content corresponding to the query keywords in each second sequence.

[0081] For example, after the TKK outputs multiple second sequences, the server 22 can retrieve the standard query content corresponding to the query keyword from the standard structured query statement based on the query keyword in each second sequence, thereby training the TKK based on the standard query content corresponding to the query keyword and the query content corresponding to the query keyword in the second sequence. Alternatively, the query keyword and the standard query content corresponding to the query keyword can be combined into a standard clause. Further, the TKK is trained based on the second sequence and the standard clause.

[0082] For example, taking the second sequence "[SELECT] document name" as an example, based on the query keyword [SELECT] in the second sequence, the standard query content corresponding to the query keyword, such as "document name", is obtained from the standard structured query statement, so that TKK is trained based on the standard query content and the query content corresponding to [SELECT] in the second sequence. Alternatively, [SELECT] and the standard query content corresponding to [SELECT], such as "document name", constitute a standard clause. Further, TKK is trained based on the second sequence and the standard clause. Similarly, TKK can be trained based on each other second sequence and the standard query content corresponding to the query keyword in each second sequence.

[0083] In this embodiment, the process from the first sequence to the second sequence, and the process of training TKK according to the second sequence and the standard query content corresponding to the query keywords in the second sequence can be recorded as the knowledge acquisition process, for example Figure 3 ② is shown. That is, during the knowledge acquisition process, the machine learning model is trained using multiple subtask learning methods, allowing the trained machine learning model to learn the knowledge corresponding to each subtask, for example, the query content corresponding to the query keywords for each subtask. This enables TKK to acquire the query content required for each subtask.

[0084] S105. Input the task prompt words, the natural language sentences, and the database model corresponding to the multiple subtasks into the machine learning model, so that the machine learning model outputs a structured query sentence corresponding to the natural language sentence.

[0085] like Figure 3 As shown in , after the knowledge acquisition process, it enters the knowledge combination process, for example Figure 3 As shown in ③, in this process, the task prompt words corresponding to each subtask, the user question, and the database model corresponding to the user question can be input into the TKK after the above training, so that TKK outputs a complete structured query statement corresponding to the user question.

[0086] S106. Retrain the machine learning model according to the structured query statement and the standard structured query statement corresponding to the natural language statement.

[0087] For example, the server 22 may retrain TKK based on the complete structured query sentence corresponding to the user question and the standard structured query sentence corresponding to the user question output by TKK.

[0088] During the knowledge acquisition process, TKK can learn the query content required for each independent subtask, but cannot learn the dependencies between the subtasks. Therefore, after TKK learns the above subtasks, its parameters will be updated. During the knowledge combination process, this embodiment can continue to fine-tune (e.g., fine-tune) the updated parameters based on the updated TKK parameters, that is, retrain. As a result, during the knowledge combination process, TKK can not only learn the relationship between the subtasks, but also learn the main task based on the subtasks it has learned. That is, based on the task prompt words corresponding to each subtask, the user question, and the database schema corresponding to the user question, it outputs the complete structured query statement corresponding to the user question. It can be understood that the input sequence of the main task is the task prompt words corresponding to each subtask, the user question, and the database schema corresponding to the user question. Among them, the task prompt words corresponding to each subtask are combined to form the task prompt words of the main task, so as to guide TKK to combine the knowledge learned in each subtask.

[0089] The disclosed embodiment divides the process of converting natural language into SQL query statements into multiple subtasks. Based on the pre-set subtasks, multiple first sequences are generated. These first sequences are input into a machine learning model to be trained, causing the machine learning model to generate multiple second sequences. Furthermore, the machine learning model is trained based on the multiple second sequences and the standard query content corresponding to the query keywords in each second sequence, allowing the machine learning model to first learn each subtask. The task prompt words corresponding to each of the multiple subtasks, the natural language statements, and the database schema are then input into the machine learning model, causing the machine learning model to output structured query statements corresponding to the natural language statements. The machine learning model is then retrained based on the structured query statements and the standard structured query statements corresponding to the natural language statements, allowing the machine learning model to learn the main task, namely, the process of converting natural language into SQL query statements. Through this task decomposition and combined learning approach, the trained machine learning model can not only accurately learn the link relationship between user questions and table column names or values, but also generate accurate SQL query statements. In addition, by decomposing, learning, and combining tasks, the machine learning model can learn the structured knowledge of SQL query statements, thereby improving the generalization ability and robustness of the machine learning model.

[0090] Figure 4 This is a flow chart of a model training method provided in another embodiment of the present disclosure. In this embodiment, the machine learning model generates multiple second sequences, including the following steps:

[0091] S401. The machine learning model generates a plurality of characters corresponding to each first sequence.

[0092] For example, for each first sequence, the machine learning model can extract multiple characters from the first sequence, and each character is recorded as a token.

[0093] Optionally, generating multiple characters corresponding to the first sequence includes: converting one or more query keywords included in the task prompt words in the first sequence into preset characters respectively; splitting the natural language sentence and the database pattern in the first sequence into one or more characters respectively, and the preset characters and the one or more characters constitute the multiple characters corresponding to the first sequence.

[0094] For example, in this embodiment, the machine learning model to be trained is the T5 model, so that the T5 model can learn the mapping from the first sequence to the second sequence. Specifically, the T5 model is based on the Encoder-Decoder framework of the Transformer and converts each first sequence into tokens.

[0095] For example, taking the first sequence "[FROM]What is the name of the document that has one of the three most common types and one of the three most common structures?;{S};{C}" as an example, [FROM] is replaced with a preset character, such as token1. In some embodiments, the preset character can also be recorded as a special marker. Further, each word in the user question "What is the name of the document that has one of the three most common types and one of the three most common structures?" is extracted, and one word is used as a token. In addition, the database pattern such as "{S};{C}" is split into multiple characters, for example, {S} is used as a token, and {C} is used as a token. Further, token1, each word in the user question, and the multiple characters corresponding to the database pattern are taken together as the multiple characters corresponding to the first sequence. Similarly, multiple characters corresponding to other first sequences can be generated. For example, for the first sequence "[GROUP_BY], [HAVING], [ORDER_BY], [LIMIT] What is the name of a document that has one of the three most common types and one of the three most common structures?; {S}; {C}", this embodiment can replace [GROUP_BY] with a preset character, i.e., a special token such as token2, replace [HAVING] with a preset character such as token3, replace [ORDER_BY] with a preset character such as token4, and replace [LIMIT] with a preset character such as token5.

[0096] S402. The machine learning model generates a second sequence corresponding to the first sequence based on the relationship features between different characters in the multiple characters.

[0097] For example, the T5 model can learn the relationship features between each token in each first sequence based on the Transformer model, thereby generating a second sequence corresponding to the first sequence.

[0098] Optionally, based on the relationship features between different characters among the multiple characters, a second sequence corresponding to the first sequence is generated, including: based on the relationship features between different characters among the multiple characters, an output sequence containing preset characters and query content is generated; and the preset characters in the output sequence are converted into query keywords corresponding to the preset characters to obtain the second sequence corresponding to the first sequence.

[0099] For example, the T5 model can learn the relationship features between each token in each first sequence based on the Transformer model, thereby generating an output sequence containing preset characters and query content. For example, based on the relationship features between each token corresponding to "[FROM] What is the name of the document that has one of the three most common types and one of the three most common structures?; {S}; {C}", the T5 model can sequentially generate tokens in the output sequence. For example, the output sequence is "token1 document collection", "token1", "document", "collection", and "collection" are respectively tokens in the output sequence. Further, the preset character "token1" in the output sequence is replaced with the query keyword corresponding to the preset character, namely [FROM], thereby obtaining a second sequence corresponding to the first sequence, and the second sequence is "[FROM] document collection".

[0100] This embodiment effectively prevents query keywords such as [GROUP_BY] or [ORDER_BY] from being recognized as two characters by converting each query keyword included in the task prompt corresponding to each subtask into a preset character. Therefore, after replacing [GROUP_BY] or [ORDER_BY] with a preset character, it is ensured that the preset character in the output sequence still refers to [GROUP_BY] or [ORDER_BY]. Furthermore, after replacing the preset character in the output sequence with [GROUP_BY] or [ORDER_BY], it is ensured that the query keyword in the second sequence remains unchanged.

[0101] In addition, the model training method provided in this embodiment can better learn the entire task from simple to complex, compared to the prior art method of directly learning the mapping relationship between natural language statements and SQL query statements. And the learning of each subtask can help the machine learning model learn the different focus points of each subtask. Ultimately, through the task combination stage, the machine learning model is more focused on generating accurate SQL statements. Through this task decomposition and combined learning method, the machine learning model not only accurately learns the link relationship between user questions and table column names or values, but also obtains more accurate SQL query statements.

[0102] In addition, by using the T5 model as the benchmark of the machine learning model, that is, using the parameters of the T5 model as the initialization parameters of the machine learning model to be trained, the common sense knowledge contained in the T5 model can be reasonably utilized. Compared with the RATSQL model that does not utilize large-scale pre-training data, this embodiment improves the generalization ability and robustness of the machine learning model by utilizing the common sense knowledge of the T5 model, as well as the decomposition, learning and combination of tasks.

[0103] The above embodiment introduces the training process of the machine learning model. When the machine learning model is trained, the trained machine learning model can be used to process natural language. In some embodiments, the process of processing natural language using the trained machine learning model can be recorded as the reasoning stage or the use stage of the machine learning model. It can be understood that the training process, reasoning stage or use stage of the machine learning model can be executed on the same device or on different devices. For example, when the server 22 completes training the machine learning model, the server 22 can use the trained machine learning model to process natural language. Alternatively, the server 22 can deploy the trained machine learning model on other servers or terminals, so that other servers or terminals use the trained machine learning model to process natural language. The following describes the method for processing natural language in conjunction with specific embodiments.

[0104] Figure 5 This is a flow chart of a natural language processing method provided by another embodiment of the present disclosure. In this embodiment, the method comprises the following specific steps:

[0105] S501: Obtain a natural language sentence.

[0106] Specifically, the implementation methods and specific principles of S501 and S101 are the same, and will not be repeated here. It is understandable that the natural language sentence in S501 and the natural language sentence in S101 can be the same user question or different user questions.

[0107] S502. Input the task prompt words corresponding to the preset multiple subtasks, the natural language sentences and the database model corresponding to the natural language sentences into the trained machine learning model, so that the machine learning model outputs the structured query sentences corresponding to the natural language sentences. The machine learning model is trained using the model training method described above.

[0108] For example, the inference phase or usage phase of the machine learning model can be similar to Figure 3 In the knowledge combination stage shown, the task prompt words corresponding to the preset multiple subtasks, the natural language sentence, and the database model corresponding to the natural language sentence are input into the trained machine learning model, so that the trained machine learning model outputs a complete structured query sentence corresponding to the natural language sentence. It can be understood that after the training process of the machine learning model is completed, it is assumed that the complete structured query sentence corresponding to the natural language sentence output by the trained machine learning model is a relatively accurate structured query sentence. The machine learning model is trained using the model training method described above, and the specific training process will not be repeated here.

[0109] Figure 6 The model training device provided by the embodiment of the present disclosure can execute the processing flow provided by the embodiment of the model training method, such as Figure 6 As shown, the model training device 60 includes:

[0110] An acquisition module 61 is used to acquire a natural language sentence;

[0111] A generating module 62 is configured to generate a plurality of first sequences according to a plurality of preset subtasks, each first sequence comprising a task prompt word corresponding to a subtask, the natural language sentence, and a database model corresponding to the natural language sentence;

[0112] an input module 63, configured to input the plurality of first sequences into a machine learning model to be trained, so that the machine learning model generates a plurality of second sequences, wherein the plurality of first sequences correspond to the plurality of second sequences in a one-to-one manner, and the second sequences include one or more query keywords and query contents corresponding to the one or more query keywords, wherein the query contents are generated based on the natural language statement and the database schema;

[0113] a training module 64 for training the machine learning model based on the plurality of second sequences and the standard query content corresponding to the query keyword in each second sequence;

[0114] The input module 63 is further configured to input the task prompt words corresponding to the plurality of subtasks, the natural language sentences, and the database model into the machine learning model, so that the machine learning model outputs a structured query sentence corresponding to the natural language sentence;

[0115] The training module 64 is further configured to retrain the machine learning model based on the structured query statement and the standard structured query statement corresponding to the natural language statement.

[0116] Optionally, the machine learning model generates multiple second sequences, including:

[0117] The machine learning model generates, for each first sequence, a plurality of characters corresponding to the first sequence;

[0118] The machine learning model generates a second sequence corresponding to the first sequence based on relationship features between different characters in the multiple characters.

[0119] Optionally, generating multiple characters corresponding to the first sequence includes:

[0120] converting one or more query keywords included in the task prompt words in the first sequence into preset characters respectively;

[0121] The natural language sentence and the database pattern in the first sequence are respectively split into one or more characters, and the preset characters and the one or more characters constitute a plurality of characters corresponding to the first sequence.

[0122] Optionally, generating a second sequence corresponding to the first sequence based on a relationship feature between different characters in the plurality of characters includes:

[0123] generating an output sequence including preset characters and query content based on relationship features between different characters in the plurality of characters;

[0124] The preset characters in the output sequence are converted into query keywords corresponding to the preset characters to obtain a second sequence corresponding to the first sequence.

[0125] Optionally, the database schema corresponding to the natural language statement includes the identifiers of one or more tables included in the database corresponding to the natural language statement, and the column names included in each of the one or more tables.

[0126] Optionally, the acquisition module 61 is further configured to: after acquiring the natural language sentence, acquire a standard structured query sentence corresponding to the natural language sentence, wherein the standard structured query sentence includes standard query contents corresponding to a plurality of query keywords.

[0127] Figure 6 The model training device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0128] Figure 7 The natural language processing device provided by the embodiment of the present disclosure can execute the processing flow provided by the embodiment of the natural language processing method, such as Figure 7 As shown, the natural language processing device 70 includes:

[0129] An acquisition module 71 is used to acquire a natural language sentence;

[0130] The input module 72 is used to input the task prompt words corresponding to the preset multiple subtasks, the natural language sentences and the database model corresponding to the natural language sentences into the trained machine learning model, so that the machine learning model outputs the structured query sentences corresponding to the natural language sentences. The machine learning model is trained using the model training method described above.

[0131] Figure 7 The natural language processing device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0132] The above describes the internal functions and structures of the natural language processing device and the model training device, which can be implemented as an electronic device. Figure 8 This is a schematic diagram of the structure of an electronic device embodiment provided by the present disclosure. Figure 8 As shown, the electronic device includes a memory 81 and a processor 82 .

[0133] The memory 81 is used to store programs. In addition to the aforementioned programs, the memory 81 may also be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method operating on the electronic device, contact data, phone book data, messages, images, videos, etc.

[0134] The memory 81 can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0135] The processor 82 is coupled to the memory 81 and executes the program stored in the memory 81 to:

[0136] Obtain natural language sentences;

[0137] Generate multiple first sequences according to the preset multiple subtasks, each first sequence including a task prompt word corresponding to a subtask, the natural language sentence, and a database model corresponding to the natural language sentence;

[0138] inputting the plurality of first sequences into a machine learning model to be trained, so that the machine learning model generates a plurality of second sequences, wherein the plurality of first sequences correspond to the plurality of second sequences in a one-to-one manner, the second sequences including one or more query keywords and query contents corresponding to the one or more query keywords, wherein the query contents are generated based on the natural language sentence and the database schema;

[0139] Training the machine learning model according to the plurality of second sequences and the standard query content corresponding to the query keyword in each second sequence;

[0140] Inputting the task prompt words corresponding to the multiple subtasks, the natural language sentences, and the database model into the machine learning model, so that the machine learning model outputs a structured query sentence corresponding to the natural language sentence;

[0141] The machine learning model is retrained according to the structured query statement and the standard structured query statement corresponding to the natural language statement.

[0142] Alternatively, the processor 82 may be configured to:

[0143] Obtain natural language sentences;

[0144] The task prompt words corresponding to the preset multiple subtasks, the natural language sentences and the database model corresponding to the natural language sentences are input into the trained machine learning model, so that the machine learning model outputs the structured query sentences corresponding to the natural language sentences. The machine learning model is trained using the model training method described above.

[0145] Further, if Figure 8 As shown, the electronic device may further include: a communication component 83, a power component 84, an audio component 85, a display 86 and other components. Figure 8 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 8 Components shown.

[0146] The communication component 83 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 83 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 83 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0147] The power supply assembly 84 provides power to various components of the electronic device. The power supply assembly 84 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0148] The audio component 85 is configured to output and / or input audio signals. For example, the audio component 85 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 81 or transmitted via the communication component 83. In some embodiments, the audio component 85 also includes a speaker for outputting audio signals.

[0149] The display 86 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0150] In addition, an embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the natural language processing method or model training method described in the above embodiment.

[0151] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0152] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A model training method, wherein: The method comprises: Obtain natural language sentences; Generate multiple first sequences according to the preset multiple subtasks, each first sequence including a task prompt word corresponding to a subtask, the natural language sentence, and a database model corresponding to the natural language sentence; inputting the plurality of first sequences into a machine learning model to be trained, so that the machine learning model generates a plurality of second sequences, wherein the plurality of first sequences correspond to the plurality of second sequences in a one-to-one manner, the second sequences including one or more query keywords and query contents corresponding to the one or more query keywords, wherein the query contents are generated based on the natural language sentence and the database schema; Training the machine learning model according to the plurality of second sequences and the standard query content corresponding to the query keyword in each second sequence; Inputting the task prompt words corresponding to the multiple subtasks, the natural language sentences, and the database model into the machine learning model, so that the machine learning model outputs a structured query sentence corresponding to the natural language sentence; The machine learning model is retrained according to the structured query statement and the standard structured query statement corresponding to the natural language statement.

2. The method according to claim 1, wherein The machine learning model generates a plurality of second sequences, including: The machine learning model generates, for each first sequence, a plurality of characters corresponding to the first sequence; The machine learning model generates a second sequence corresponding to the first sequence based on relationship features between different characters in the multiple characters.

3. The method according to claim 2, wherein: Generating a plurality of characters corresponding to the first sequence includes: converting one or more query keywords included in the task prompt words in the first sequence into preset characters respectively; The natural language sentence and the database pattern in the first sequence are respectively split into one or more characters, and the preset characters and the one or more characters constitute a plurality of characters corresponding to the first sequence.

4. The method according to claim 3, wherein: Generating a second sequence corresponding to the first sequence based on a relationship feature between different characters in the plurality of characters includes: generating an output sequence including preset characters and query content based on relationship features between different characters in the plurality of characters; The preset characters in the output sequence are converted into query keywords corresponding to the preset characters to obtain a second sequence corresponding to the first sequence.

5. The method according to claim 1, wherein The database schema corresponding to the natural language statement includes the identifiers of one or more tables included in the database corresponding to the natural language statement, and the column names included in each of the one or more tables.

6. The method according to claim 1, wherein After obtaining the natural language sentence, the method further includes: A standard structured query statement corresponding to the natural language statement is obtained, where the standard structured query statement includes standard query contents corresponding to a plurality of query keywords.

7. A natural language processing method, wherein: The method comprises: Obtain natural language sentences; The task prompt words corresponding to the preset multiple subtasks, the natural language sentences and the database model corresponding to the natural language sentences are input into the trained machine learning model, so that the machine learning model outputs the structured query sentences corresponding to the natural language sentences. The machine learning model is trained using the method described in any one of claims 1-6.

8. A model training device, wherein: The device comprises: An acquisition module, used to acquire natural language sentences; A generating module, configured to generate a plurality of first sequences according to a plurality of preset subtasks, each first sequence comprising a task prompt word corresponding to a subtask, the natural language sentence, and a database model corresponding to the natural language sentence; an input module, configured to input the plurality of first sequences into a machine learning model to be trained, so that the machine learning model generates a plurality of second sequences, wherein the plurality of first sequences correspond to the plurality of second sequences in a one-to-one manner, the second sequences comprising one or more query keywords and query content corresponding to the one or more query keywords, wherein the query content is generated based on the natural language statement and the database schema; a training module, configured to train the machine learning model based on the plurality of second sequences and standard query content corresponding to the query keywords in each second sequence; The input module is further configured to input the task prompt words corresponding to the plurality of subtasks, the natural language sentences, and the database model into the machine learning model, so that the machine learning model outputs a structured query sentence corresponding to the natural language sentence; The training module is also used to retrain the machine learning model based on the structured query statement and the standard structured query statement corresponding to the natural language statement.

9. A natural language processing device, wherein: include: An acquisition module, used to acquire natural language sentences; An input module is used to input task prompt words corresponding to multiple preset subtasks, the natural language sentences and the database model corresponding to the natural language sentences into a trained machine learning model, so that the machine learning model outputs a structured query sentence corresponding to the natural language sentence. The machine learning model is trained using the method described in any one of claims 1 to 6.

10. An electronic device, wherein: include: Memory; processor; as well as computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method according to any one of claims 1 to 7.

11. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images