Model training and task execution method and device, electronic equipment, storage medium and computer program product

Abstract

The invention discloses a model training and task execution method and device, electronic equipment, a storage medium and a computer program product, and the method comprises the steps: carrying out the pre-training of a plurality of tasks on a first model based on a training sample, and obtaining a second model; obtaining a natural language (NL) problem, and calling a second model to process the NL problem to obtain a database query statement corresponding to the NL problem; executing the database query statement to output a query result corresponding to the NL question; wherein the training samples are generated based on database query codes and corresponding code annotations, and the tasks comprise an identifier sensing task and a bimodal learning task of natural languages and codes; the first model is used for converting the NL problem into a database query code.

Description

Technical Field

[0001] The present application relates to the field of wireless technology, and in particular to a model training and task execution method, device, electronic device, storage medium and computer program product. Background Art

[0002] The natural language to structured query language conversion task aims to convert natural language problems in the database field into structured query languages ​​that can be executed in relational databases, thereby improving the usability of relational databases. The input of the conversion model used to perform this conversion task is mostly a natural language question and a database model, and the output is the structured query language (SQL) query statement corresponding to the natural language question. However, the relevant conversion model cannot accurately grasp the entity association between the input statement and the output statement during the task generation process, resulting in low entity recognition accuracy and weak generalization ability of the conversion model. Summary of the invention

[0003] To solve related technical problems, the embodiments of the present application provide a model training and task execution method, device, electronic device, storage medium and computer program product.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application embodiment provides a model training method, including:

[0006] Based on the training samples, the first model is pre-trained on multiple tasks; wherein,

[0007] The training samples are generated based on database query codes and corresponding code comments, and the multiple tasks include identifier perception tasks and bimodal learning tasks of natural language and code; the first model is used to convert natural language (NL) questions into database query codes.

[0008] In the above scheme, the method further comprises:

[0009] Based on the set relationship matrix, the first model completed by pre-training is adjusted; wherein,

[0010] The set relationship matrix represents the relationship between NL problem entities and / or databases.

[0011] In the above solution, the training samples also include:

[0012] Contextual information of the code comments corresponding to the database query code.

[0013] In the above scheme, pre-training the first model for the identifier perception task includes:

[0014] Training the first model for a first task, wherein the first task is to predict the mask span of the database query code in the training sample;

[0015] Performing a second task training on the first model that has completed the first task training, where the second task is used to mark whether a token of a database query code in a training sample is an identifier;

[0016] The first model that has completed the second task training is trained for a third task, where the third task is used to predict an identifier masked by a database query code in a training sample.

[0017] In the above scheme, the method further comprises:

[0018] Before training the first model trained for the second task for the third task, all identifiers of the database query code in the training sample are masked using the same token.

[0019] In the above scheme, the first model is trained for the dual-modal learning task of natural language and code, including:

[0020] training the first model for a fourth task, the fourth task being to convert natural language into code; and,

[0021] The first model is trained on a fifth task, wherein the fifth task is to convert the code into natural language.

[0022] In the above solution, the relationship between the NL problem entities and / or databases includes one or more of the following:

[0023] The relationship between NL problem entities;

[0024] The relationship between database tables;

[0025] The relationships between database columns;

[0026] The relationship between database tables and columns;

[0027] NL Question The relationship between entities and database tables and / or database columns.

[0028] In the above scheme, the encoder in the first model includes a semantic encoder and a relational encoder, and the output of the semantic encoder serves as the input of the relational encoder; wherein,

[0029] The relation encoder is used to encode the first relation graph as bias information of the self-attention matrix, and the first relation graph is used to characterize the relationship between each natural language question entity node and the database table name node and the database column name node.

[0030] In the above scheme, the decoder in the first model includes a first attention layer and a second attention layer; wherein,

[0031] The output information of the encoder in the first model is connected to the database query code generated by the decoder as the input of the first attention layer; the output information of the encoder in the first model is used as the key and value used by the second attention layer, and the previous output information of the decoder is used as the query item used by the second attention layer.

[0032] In the above scheme, the method further comprises:

[0033] Based on the set relationship matrix, the attention score is calculated in the decoder.

[0034] The present application provides a task execution method, including:

[0035] Get NL questions;

[0036] Calling a second model to process the NL problem to obtain a database query statement corresponding to the NL problem; the second model is trained based on any of the above model training methods;

[0037] Execute the database query statement to output the query result corresponding to the NL question.

[0038] The present application also provides a model training device, including:

[0039] A training unit is used to pre-train the first model for multiple tasks based on the training samples; wherein,

[0040] The training samples are generated based on database query codes and corresponding code comments, the multiple tasks include identifier awareness tasks and bimodal learning tasks of natural language and code; the first model is used to convert NL questions into database query codes.

[0041] The embodiment of the present application also provides a task execution device, including:

[0042] An acquisition unit, used to acquire NL questions;

[0043] A calling unit, configured to call a second model to process the NL problem and obtain a database query statement corresponding to the NL problem; the second model is trained based on any of the above-mentioned model training methods;

[0044] An execution unit is used to execute the database query statement to output the query result corresponding to the NL question.

[0045] The embodiment of the present application further provides a first electronic device, comprising: a first processor and a first communication interface; wherein,

[0046] The first processor is used to pre-train the first model for multiple tasks based on the training samples; wherein,

[0047] The training samples are generated based on database query codes and corresponding code comments, the multiple tasks include identifier awareness tasks and bimodal learning tasks of natural language and code; the first model is used to convert NL questions into database query codes.

[0048] The embodiment of the present application further provides a second electronic device, including: a second processor and a second communication interface; wherein,

[0049] The second processor is configured to:

[0050] Get NL questions;

[0051] Calling a second model to process the NL problem to obtain a database query statement corresponding to the NL problem; the second model is trained based on any of the above model training methods;

[0052] Execute the database query statement to output the query result corresponding to the NL question.

[0053] The present application also provides an electronic device, including: a processor and a memory for storing a computer program that can be run on the processor.

[0054] Wherein, when the processor is used to run the computer program, it executes the steps of any of the above-mentioned model training methods, or executes the steps of any of the above-mentioned task execution methods.

[0055] An embodiment of the present application also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned model training methods are implemented, or the steps of any of the above-mentioned task execution methods are implemented.

[0056] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned model training methods, or implements the steps of any of the above-mentioned task execution methods.

[0057] In the model training and task execution method, device, electronic device, storage medium and computer program product provided by the embodiments of the present application, in the process of training the model for converting NL problems into database query codes, the model is pre-trained for multiple tasks based on the training samples generated by the database query code and the corresponding code annotations, and the multiple tasks include identifier perception tasks and bimodal learning tasks of natural language and code, which can better improve the identifier perception ability of the model and enable the model to better understand natural language and code language, so that the trained model has higher accuracy in entity recognition and stronger generalization ability. In actual application, the NL problem is processed based on the trained model to obtain the corresponding database query statement, and more accurate query results can be output by executing the database query statement. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A flowchart of a method for converting natural language to structured query language in the related art;

[0059] Figure 2 A schematic diagram of a model training method flow in an embodiment of the present application;

[0060] Figure 3 This is a flow chart of another model training method according to an embodiment of the present application;

[0061] Figure 4 This is a schematic diagram of the relationship between NL problem entities in an embodiment of the present application;

[0062] Figure 5 This is a schematic diagram of the relationship between database tables / columns in an embodiment of the present application;

[0063] Figure 6 This is a schematic diagram of the relationship between an NL problem entity and a database table / column according to an embodiment of the present application;

[0064] Figure 7 This is a schematic diagram of a relationship matrix according to an embodiment of the present application;

[0065] Figure 8 A schematic diagram of a model architecture of an embodiment of the present application;

[0066] Fig. 9 This is a schematic diagram of a model training device according to an embodiment of the present application;

[0067] Fig.10 This is a schematic diagram of a task execution device according to an embodiment of the present application;

[0068] Fig.11 This is a schematic diagram of a first electronic device according to an embodiment of the present application;

[0069] Fig.12 This is a schematic diagram of a second electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0070] Relational databases store rich structured data. Users can accurately obtain target resources by querying relational databases. However, the operation of relational databases needs to be implemented based on SQL, and the threshold for resource acquisition is high. The natural language to structured query language (NL2SQL) task aims to convert natural language problems in the database field into executable structured query languages ​​in relational databases, thereby greatly improving the usability of relational databases. Figure 1 The execution framework of the NL2SQL task is shown. Specifically, the input of the NL2SQL task is mostly a natural language question and a database schema, and the output is the SQL query statement corresponding to the natural language question. For example, the natural language question can be "What's the total number of songs originally performed by anna nalick?", and the corresponding SQL query statement is "SELECT COUNT Song choice WHERE Original artist = anna nalick". The goal of the NL2SQL task is to build a model that can understand natural language questions and map the natural language questions to SQL query statements of the corresponding database, so as to retrieve information from the database or perform specific operations. The constructed model needs to be able to understand the semantics of natural language questions, including parsing the entities, attributes, conditions, relations, etc. of natural language questions. It also needs to be able to understand the structure of a given database so that it can correctly map the entities in the natural language questions (NL question entities) to the corresponding tables or columns in the database schema. Among them, the entity in the NL problem includes key entities, such as time, number, product, etc. For example, in the query "Find all products with sales exceeding 10 million in 2023", "2023", "10 million" and "product" are key entities; the entity also includes an operation entity, which represents the operation the user wants to perform, such as "find", "exceed", "sort in descending order", etc.

[0071] In recent years, the mainstream NL2SQL task processing method is based on the Seq2Seq framework. Seq2Seq is a neural network structure based on a sequence-to-sequence model, generally based on the Transformer architecture, and consists of two parts: an encoder and a decoder. Among them, the encoder is responsible for information encoding, thereby encoding the input natural language questions, database schemas and other information into contextual information that the model can understand. The decoder is responsible for decoding the representation information output by the encoder and finally obtaining SQL statements. In addition, in order to improve the accuracy of entity alignment, in related technologies, the association information of natural language questions and database schemas is usually put into the encoder to obtain contextual representation information with entity association information. Among them, Graphix-T5 is used as a model to solve the NL2SQL task. The pre-trained model of this model is the T5 model, and a relationship graph containing natural language question nodes and database schema nodes is modeled. The relationship information is integrated into the self-attention mechanism as a bias, so that the representation information obtained by the encoder is contextual information with entity association, which improves the performance of generating accurate SQL statements from natural language queries.

[0072] However, the T5 model has a relatively limited dataset, which results in the insufficient generalization ability of the Graphix-T5 model in completing downstream tasks of structured generation. Moreover, the Graphix-T5 model cannot accurately grasp the entity associations between the input and output sentences in the generation task, resulting in entity recognition errors in the output. Therefore, when structured sentence generation is used as a downstream task, the Graphix-T5 model cannot perform the generation task well when encountering more complex samples. Moreover, since the Graphix-T5 model only retains the association information between natural language questions and database schemas in the encoder, it only obtains contextual information during encoding, does not further explore entity relationships during decoding, and does not provide substantial guidance information during decoding, resulting in the model being unable to accurately identify and link entities in the generation task of complex samples, thereby reducing the accuracy of the model.

[0073] In the model training and task execution method, device, electronic device, storage medium and computer program product provided by the embodiments of the present application, in the process of training the model for converting NL problems into database query codes, the model is pre-trained for multiple tasks based on the training samples generated by the database query code and the corresponding code annotations, and the multiple tasks include identifier perception tasks and bimodal learning tasks of natural language and code, which can better improve the identifier perception ability of the model and enable the model to better understand natural language and code language, so that the trained model has higher accuracy in entity recognition and stronger generalization ability. In actual application, the NL problem is processed based on the trained model to obtain the corresponding database query statement, and more accurate query results can be output by executing the database query statement.

[0074] The present application embodiment provides a model training method, referring to Figure 2 , the method comprising:

[0075] Step 201: Based on the training samples, pre-train the first model for multiple tasks.

[0076] The training samples are generated based on database query codes and corresponding code comments, the multiple tasks include identifier awareness tasks and bimodal learning tasks of natural language and code; the first model is used to convert NL questions into database query codes.

[0077] In an embodiment of the present application, the first model is used to convert NL questions into database query codes. In order to improve the conversion capabilities of the first model, such as generalization capabilities, the first model needs to be pre-trained for multiple tasks.

[0078] In an embodiment of the present application, pre-training of the first model for multiple tasks is performed based on a training sample, which can also be referred to as a data set. During pre-training, the training sample or data set can be divided into a training set, a validation set, or a test set. The training sample can be generated based on a large amount of rich database query code and corresponding code comments. Among them, the database query code can use the code of the open source code repository, and can include multiple programming languages, such as C, Java, Python, JavaScript; the corresponding code comments can help the first model better understand the relationship between the NL query and the code, and improve the performance of the first model during pre-training. For example, the code comments can help the first model understand why a specific table or column is selected, or why a specific function is used.

[0079] In one embodiment, the training sample further includes:

[0080] Contextual information of the code comments corresponding to the database query code.

[0081] In an embodiment of the present application, contextual information, such as business logic, data constraints, or specific query requirements, may also be provided in the code annotations corresponding to the database query code. This helps the first model to better understand the semantics of the NL query and accurately map the semantics of the NL query to the corresponding query code, so that the first model can learn a wider range of semantic knowledge and patterns, which helps the first model to show better generalization capabilities when processing new data and new scenarios. For example, the first model can learn the common features between different database models, so that it can adapt to the new database structure more easily.

[0082] In an embodiment of the present application, the first model is pre-trained for multiple tasks, wherein the multiple tasks include identifier perception tasks and bimodal learning tasks of natural language and code. For the identifier perception task, the identifier perception ability needs to be used as the training goal in the pre-training of the first model. The identifier perception ability not only helps the first model to accurately map the vocabulary in the NL query to the corresponding identifier in the code, but also helps the first model to maintain logical consistency when generating query statements. For example, when a table or column is mentioned in the NL query, the model needs to be able to recognize and correctly reference these tables or columns in the query statement, and correctly express these logical relationships in the query statement. Performing bimodal learning tasks of natural language and code can enable the model to better understand natural language and code language, so that the trained model has higher accuracy and stronger generalization ability in entity recognition.

[0083] In one embodiment, pre-training the first model for an identifier-aware task includes:

[0084] Training the first model for a first task, the first task being used to predict mask spans of database query codes in training samples;

[0085] Performing a second task training on the first model that has completed the first task training, where the second task is used to mark whether a token of a database query code in a training sample is an identifier;

[0086] The first model that has completed the second task training is trained for a third task, where the third task is used to predict an identifier masked by a database query code in a training sample.

[0087] In an embodiment of the present application, the first model is pre-trained for multiple tasks based on training samples, wherein, in actual applications, the training samples often contain noise or inaccurate information. By performing denoising training on the first model during the pre-training of the identifier perception task, the first model can better learn to extract useful information in a noisy environment, thereby improving the robustness of the first model.

[0088] In an embodiment of the present application, when performing denoising training, the first model is trained for the first task, and the first task is used to predict the mask span of the database query code in the training sample. Specifically, the masked span prediction (MSP, Masked Span Prediction) method can be used to train the first task. The core idea of ​​the method is to randomly mask one or more continuous text or code spans in the input data, and then require the model to predict these masked spans in combination with sentinel tokens at the decoder, that is, the first task is used to predict the mask span of the database query code in the training sample. In the masked span prediction, spans of any length can be randomly masked. However, in practical applications, in order to balance training efficiency and model performance, a range of span lengths can also be set, and the length is randomly selected within the range for masking. Among them, the sentinel token is used to indicate which masked spans the model should pay attention to when generating the output. By introducing the sentinel token at the decoder, the model can more easily identify and restore the masked spans.

[0089] In an embodiment of the present application, when performing denoising training, the first model that has completed the first task training is trained for a second task, and the second task is used to mark whether the token of the database query code in the training sample is an identifier. Specifically, in this denoising training method, the key to marking whether the token of the database query code in the training sample is an identifier is to convert the identifier recognition task into a binary classification problem, that is, to determine whether a given token belongs to the identifier category. Exemplarily, in the task of predicting whether a token in the code is an identifier, a binary cross entropy loss function can be used as the loss function of the model to measure the difference between the probability distribution predicted by the first model and the probability distribution of the true label.

[0090] In an embodiment of the present application, when performing denoising training, the first model that has completed the second task training is trained for a third task, and the third task is used to predict the identifiers masked by the database query code in the training sample. Specifically, in the denoising training method, before predicting the identifiers masked by the database query code in the training sample, all identifiers in the database query code can be masked (or called masking). In order to balance the training efficiency and model performance, some identifiers can also be selectively masked.

[0091] In one embodiment, the method further comprises:

[0092] Before training the first model that has completed the training for the second task for the third task, all identifiers of the database query code in the training sample are masked using the same token.

[0093] In an embodiment of the present application, before the first model that has completed the training of the second task is trained for the third task, the same token can be used for all identifiers of the database query code in the training sample, that is, a unique token is assigned to each different identifier at the same time, and these tokens can be correctly mapped back to the original identifier during decoding, which will not affect the code semantics, so that the first model can more easily distinguish different identifiers. In addition, the allocation of the same token can be cross-context, that is, the same identifier appearing in different code segments can be assigned the same unique token. After masking the identifiers in the code, the unique tokens and sentinel tokens of different identifiers can be arranged in the target sequence and then the masked identifier prediction can be performed.

[0094] In an embodiment of the present application, denoising training is performed in the pre-training of the first model for the identifier perception task in the above manner, which can improve the identifier perception ability of the first model, help the first model to cope with variable and complex samples, and help enhance the first model's ability to understand database query statements.

[0095] In one embodiment, training the first model for a bimodal learning task of natural language and code includes:

[0096] The first model is trained on a fourth task, the fourth task being used to convert natural language into code; and the first model is trained on a fifth task, the fifth task being used to convert code into natural language.

[0097] In an embodiment of the present application, in order to narrow the difference between the learning objectives of the pre-training stage and the fine-tuning stage, in the pre-training stage, bimodal learning of natural language and code language can be used as the training objective, and not only the conversion of natural language into code is trained as the fourth task, but also the conversion of code into natural language is trained as the fifth task, that is, natural language and code language can be bidirectionally converted, and the bidirectional conversion can be regarded as a dual task. Specifically, during training, for each pair of bimodal data, training can be performed by constructing two training instances with opposite conversion directions and adding language identifiers. Among them, bimodal data refers to a data set with a corresponding relationship between natural language and code language (such as SQL language).

[0098] In an embodiment of the present application, by training the first model on the fourth task and the fifth task of the dual-modal learning task of natural language and code, the ability of the first model in aligning natural language and code language can be improved, which helps the first model to better adapt to subsequent adjustments to the first model.

[0099] In one embodiment, the method further comprises:

[0100] Based on the set relationship matrix, the first model completed by pre-training is adjusted; wherein,

[0101] The set relationship matrix represents the relationship between NL problem entities and / or databases.

[0102] In the embodiment of the present application, although the first model after pre-training has certain generalization ability and NL problem entity recognition ability, the entity alignment ability of the first model needs to be enhanced. Among them, entity alignment refers to the process of correctly mapping entities in natural language (such as table names, column names) to corresponding elements in the database. Since natural language expressions are diverse and code languages ​​are relatively fixed, it is necessary to enhance the entity alignment ability of the first model. For this reason, the pre-trained first model can be adjusted based on the set relationship matrix to enhance the entity alignment ability of the first model, wherein the set relationship matrix represents the relationship between NL problem entities and / or databases.

[0103] In one embodiment, the relationship between the NL problem entity and / or the database includes one or more of the following:

[0104] The relationship between NL problem entities;

[0105] The relationship between database tables;

[0106] The relationships between database columns;

[0107] The relationship between database tables and columns;

[0108] NL Question The relationship between entities and database tables and / or database columns.

[0109] In the embodiment of the present application, the relationship between NL question entities refers to the relationship between natural language entities. For example, for the NL question "find the number of pet dogs raised by female students", "female" is a modifier of "students", and there is a modification relationship between "female" and "students". This type of relationship information can be obtained by using syntax trees and dependency syntax analysis techniques to reveal the syntactic structure of natural language questions and the position of each NL question entity in the question and the semantic role it plays, helping the model to better understand the meaning and structure of the question.

[0110] In the embodiment of the present application, the relationship between NL problem entities and / or databases includes the relationship between tables in the database, the relationship between columns in the database, and the relationship between tables and columns in the database. Specifically, the relationship between tables in the database can be the relationship between primary keys and foreign keys between tables, such as a one-to-one relationship, a one-to-many relationship, or a many-to-many relationship; the relationship between columns in the database can be the relationship between two columns in the same table; the relationship between tables and columns in the database can be the relationship that a certain column is included in a certain table.

[0111] In the embodiment of the present application, the relationship between the NL problem entity and / or the database includes the relationship between the NL problem entity and the table of the database and / or the column of the database. For example, for the NL problem "find the number of pet dogs raised by female students", "pet" can be exactly matched (Exact-match) to the "pet" table in the database; "female" can be equivalently matched (Value-match) to the "gender" column in the "student" table in the database; "students" can be partially matched (Partial-match) to the "student" table in the database.

[0112] In an embodiment of the present application, by adjusting the pre-trained first model based on the relationship between NL problem entities and / or databases, the accuracy of selecting database table names and column names when generating SQL statements can be improved, which helps the first model to better understand the meaning and structure of NL problems, the organization and association of databases, and the correspondence between NL problems and database models.

[0113] In one embodiment, the encoder in the first model includes a semantic encoder and a relational encoder, and the output of the semantic encoder serves as the input of the relational encoder.

[0114] Among them, the relationship encoder is used to encode the first relationship graph as the bias information of the self-attention matrix, and the first relationship graph is used to characterize the relationship between each natural language question entity node and the database table name node and the database column name node.

[0115] In an embodiment of the present application, the first model includes an encoder and a decoder. The first model may include one or more encoders, and may also include one or more decoders. Among them, the encoder in the first model includes a semantic encoder and a relational encoder. When constructing a semantic encoder, the original encoder in the T5 model can be used to encode natural language problems and database models, or the encoder parameters in the T5 model can be used to initialize other parameters; when constructing a relational encoder, the Relational Graph Attention Network (RGAT) can be used to process data containing relational information. When constructing an association between a semantic encoder and a relational encoder, the output of the semantic encoder can be used as the input of the relational encoder. Furthermore, the semantic encoder and the relational encoder can also be combined to construct a dual encoder of semantics and relations.

[0116] In an embodiment of the present application, when constructing a relational encoder, before processing data containing relational information in the form of a relational graph attention network, the relationship between NL problem entity nodes and database table name nodes and database column name nodes can be mined to facilitate the construction of the first relational graph. Among them, the nodes of the first relational graph include NL problem entity nodes and database model nodes, the database model nodes include database table name nodes and database column name nodes, and the edges of the first relational graph are the relationships between these nodes. In order to obtain the relationship between database model nodes, the database model can be directly accessed; in order to obtain the relationship between NL problem entity nodes, it can be obtained through a syntactic analyzer; in order to obtain the relationship between NL problem entity nodes and database model nodes, it can be obtained by string comparison.

[0117] In the embodiment of the present application, after the construction of the first relationship graph is completed, the first relationship graph can be encoded as the bias information of the self-attention matrix, that is, the first relationship graph is integrated into the encoder for encoding. In this way, the encoder can obtain context representation information, which carries the relationship information in the first relationship graph, and the context representation information can be used as the output of the encoder.

[0118] In an embodiment of the present application, an encoder including a semantic encoder and a relational encoder is introduced into the first model, and the output of the semantic encoder is used as the input of the relational encoder, wherein the relational encoder is used to encode the first relationship graph as bias information of the self-attention matrix, and the first relationship graph is used to characterize the relationship between each natural language question entity node and the database table name node, as well as the database column name node. The relationship information between the natural language question entity nodes and the database table name nodes, as well as the database column name nodes can be integrated into the encoder, thereby enhancing the entity recognition and entity alignment capabilities of the model.

[0119] In one embodiment, the decoder in the first model includes a first attention layer and a second attention layer; wherein,

[0120] The output information of the encoder in the first model is connected to the database query code generated by the decoder as the input of the first attention layer; the output information of the encoder in the first model is used as the key and value used by the second attention layer, and the previous output information of the decoder is used as the query item used by the second attention layer.

[0121] In the embodiment of the present application, the decoder in the first model includes a first attention layer. The output information of the encoder in the first model includes relationship information, and the output information of the encoder in the first model can be connected with the database query code generated by the decoder as the input of the first attention layer. Specifically, the output information of the encoder in the first model can be used as a prefix and connected with the database query code (such as an SQL statement) generated by the decoder as the input of the first attention layer.

[0122] In the embodiment of the present application, the decoder in the first model also includes a second attention layer. The output information of the encoder in the first model includes relationship information, and the output information of the encoder in the first model is used as the key (K, Key) and value (V, Value) used by the second attention layer, and the previous output information of the decoder is used as the query item (Q, Query) used by the second attention layer.

[0123] In an embodiment of the present application, the above-mentioned first attention layer and second attention layer are introduced by the decoder in the first model, and the output information of the encoder in the first model is connected with the database query code generated by the decoder as the input of the first attention layer. The output information of the encoder can be used to assist the decoder in better associating NL problems and database structures when generating SQL tokens, thereby improving the accuracy of generating database query statements; by using the output information of the encoder in the first model as the key and value used by the second attention layer, and using the previous output information of the decoder as the query item used by the second attention layer, the relationship information can be used at different attention layers to promote information interaction between different attention layers, thereby improving the accuracy of generating database query statements.

[0124] In one embodiment, the method further comprises:

[0125] Based on the set relationship matrix, the attention score is calculated in the decoder.

[0126] In an embodiment of the present application, since the input parts of the first attention layer and the second attention layer in the decoder both include the output information of the encoder in the first model, and the output information of the encoder in the first model includes relationship information, in the calculation process of these two attention layers, the relationship graph attention network can be used to process the data containing relationship information, that is, based on the set relationship matrix, the relevant attention scores are calculated in the relationship graph attention network.

[0127] In an embodiment of the present application, by calculating the attention score in the decoder based on the set relationship matrix, the decoder can more effectively utilize the relationship information, helping the first model to more accurately understand the relationship between the NL problem and the database structure, thereby improving the relevance and accuracy of the generated database query statement.

[0128] Based on the model training method provided in the above embodiment, Figure 3 The schematic diagram of the principle of the model training scheme of the embodiment of the present application is shown, combined with Figure 3 When the embodiment of the present application performs model training, first, in the pre-training stage, on a rich code data set, a multi-task learning method is used to build a pre-training model with identifier perception, code and natural language bimodal learning as the training goals, which effectively improves the generalization ability of the model in terms of entity alignment ability; secondly, in the fine-tuning stage, the relationship information between natural language questions and database patterns is modeled and integrated into the calculation of the Self-Attention module of the encoder and decoder, completing the fusion of the encoder and decoder with the relationship information, so that the model can better identify and link entities.

[0129] Based on the model training method provided in the above embodiment, the embodiment of the present application also provides a task execution method, which includes:

[0130] Get NL questions;

[0131] Calling a second model to process the NL problem to obtain a database query statement corresponding to the NL problem; the second model is trained based on the model training method provided in the above embodiment;

[0132] Execute the database query statement to output the query result corresponding to the NL question.

[0133] In the embodiment of the present application, based on the trained first model, a second model is obtained. The second model is used to convert the acquired NL question into a database query code or statement, and then by executing the database query statement, the query result corresponding to the NL question can be output. Through the above steps, a second model for natural language to SQL language conversion is constructed, which improves the ease of use of relational database queries and alleviates the problem of difficulty in data query due to the user's inability to master SQL technology.

[0134] The present application is further described in detail below in conjunction with application examples.

[0135] In order to construct a conversion model for accurate conversion from natural language to SQL statements, the training of the model in the application embodiment of the present application mainly includes stage one (pre-training stage) and stage two (fine-tuning stage).

[0136] Phase 1: Pre-training phase.

[0137] Step 1-1: Selection of training samples.

[0138] In the scenario of this embodiment, the training sample can also be described as a data set or a code data set. In the selection of training samples, the present application can use code data from an open source code repository for training, and can cover a variety of programming languages, including C, Java, Python, JavaScript, etc. The data set contains code snippets, corresponding comments, and contextual information in actual projects. Among them, the combination of code snippets and comments can help the model learn the common representation of natural language and code.

[0139] Rich training samples help the model cope with the varied and complex samples in generation tasks, and the selection of code data sets can help the model learn the characteristics of structured sentence generation in the pre-training stage.

[0140] Step 1-2: Pre-training tasks using identifier awareness and bimodal learning of natural language and code language.

[0141] The model uses multi-task learning in the pre-training stage, and the main training objectives include identifier perception and dual-modal learning of natural language and code language. Multi-task learning further enhances the model's ability to understand and generate code, especially identifier perception and dual-modal learning of code and natural language help the model improve its entity recognition ability, so that it can better adapt to downstream NL2SQL tasks, thus improving the generalization ability of the model.

[0142] In order to improve the identifier perception capability of the model, this embodiment takes identifier perception as the training goal and pre-trains the model using the denoising method in steps 1-3 to 1-5.

[0143] Step 1-3: Perform the masking span prediction task in denoising pre-training.

[0144] The masked span prediction task is to use the span mask target to randomly mask spans of arbitrary length, and combine the sentinel tokens at the decoder to predict these mask spans. In specific applications, the loss function can be used as follows:

[0145]

[0146] in, is the loss function, and θ is the model parameter. θ represents the probability calculated by the model parameter θ, x \mask is the mask input, x mask is the mask sequence predicted from the decoder, t represents the number of tokens generated at the current moment, and k represents x mask The total number of tokens in Indicates the token generated at the current moment. is the span sequence that has been generated up to the current moment. The meaning of this formula is that for each known token (i.e., the token of the non-masked position) in the generated sequence, the negative of the logarithmic probability it generates is calculated, and these values ​​are added as the loss.

[0147] Step 1-4: Perform the identifier labeling task in denoising pre-training.

[0148] The identifier tagging task refers to modeling the task as a sequence labeling problem to predict whether a token in the code is an identifier. In specific applications, the loss function can use the binary cross entropy loss function, which is expressed as follows:

[0149]

[0150] in, is the loss function, θ e is the encoder parameter of the model. m represents the total number of tokens in the code, i represents the position of the current token, and y i Indicates whether the current token is an identifier, represented by 0 or 1, p i Indicates the probability that the current token is an identifier.

[0151] Step 1-5: Perform the masked identifier prediction task in denoising pre-training.

[0152] The masked identifier prediction task is to mask all identifiers in the code, and use unique tokens for all occurrences of different identifiers, which will not affect the code semantics. Then, the unique tokens and sentinel tokens of different identifiers are arranged in the target sequence for prediction. Specifically, the prediction can be performed in an autoregressive manner, and the following loss function is defined:

[0153]

[0154] in, is the loss function, θ is the model parameter. I represents the label sequence, |I| represents the length of the label sequence, j represents the position of the current identifier, P θ represents the probability calculated by the model parameter θ, x \I represents the input sequence that masks the labeled sequence, I <j Represents the first j-1 annotations of the annotation sequence.

[0155] Furthermore, in order to narrow the difference between the learning objectives of the pre-training stage and the fine-tuning stage and improve the model's ability to align natural language and code language, this embodiment uses natural language and code language bimodal learning as the training goal, so that natural language and code language can be converted in both directions, which helps the model adapt to downstream tasks. Specifically, this embodiment regards the conversion from natural language to code language and the conversion from code language to natural language as dual tasks, performs model optimization, constructs two training instances with opposite conversion directions for each pair of bimodal data, and adds language identifiers.

[0156] After the pre-training phase, a fine-tuning phase can be performed.

[0157] Phase 2: Fine-tuning phase.

[0158] Step 2-1: Model the relationship information between natural language questions and database schema.

[0159] In order to improve the accuracy of selecting database table and column names when generating SQL statements, it is necessary to mine the relationship between NL question entities and tables / columns in the database schema, and model the relationship information between natural language questions and database schemas. The main relationship information is divided into three categories, namely:

[0160] The first category is the relationship between NL problem entities. Figure 4As shown in the figure, for the NL question "Find the number of dog pets that are raised by female students", "female" is the modifier of "students". This kind of relationship information needs to be obtained using syntax trees and dependency parsing techniques to reveal the syntactic structure of natural language questions and the position and semantic role of each NL question entity in the question, helping the model to better understand the meaning and structure of the question.

[0161] The second category is the relationship between database tables / columns. Figure 5 For example, the "PetAge" column belongs to the "Pets" table, and there is a foreign key relationship between the "Pets" table and the "Has_Pet" table. This type of relationship is directly obtained from the database schema. The main relationships include primary key and foreign key relationships, the relationship that a column is contained in a table, and the relationship between two columns in the same table. This information provides the model with a basic understanding of the database structure and helps the model better understand the organization and association of the database.

[0162] The third category is the relationship between NL problem entities and database tables / columns. Figure 6 As shown in the figure, for example, there is an exact match relationship for "pets" and an equivalent match relationship for "female" and "sex". This type of relationship information is an implicit relationship, including the exact match and partial match relationship between the NL question entity and the database pattern item, as well as the consistent relationship between the NL question entity and a cell value in the database. This relationship is very important for the model to understand the correspondence between natural language questions and database patterns.

[0163] After obtaining the above three types of relationships, we use the form of a graph to store the relationship information. The nodes of the graph include natural language question nodes and pattern nodes (including database table name nodes and database column name nodes). The edges of the graph are the three types of relationships mentioned above, and each specific relationship is represented by a vector. Taking the natural language question and database pattern above as an example, the relationship matrix is ​​given, such as Figure 7 shown.

[0164] Step 2-2: Incorporate relational information into the calculation of the self-attention modules of the encoder and decoder.

[0165] like Figure 8As shown in the model architecture, by modeling the relationship information between NL problem entities and database table / column names and integrating it into the encoder and decoder of the model, the model with fused relationship information is fine-tuned based on the NL2SQL task, which improves the accuracy of the model's NL2SQL task, especially the accuracy of entity alignment. By accurately capturing and integrating relationship information, the accuracy and efficiency of converting natural language queries into SQL statements are improved, further enhancing the system's intelligent processing capabilities.

[0166] Specifically, for the dual encoder of semantics and relations in the model architecture, the dual encoder is responsible for encoding the combined information of semantics and relations, which can be completed by the semantic encoding block and the relation information encoding block respectively. The semantic encoding block is completed by the original encoder of T5, which is responsible for encoding natural language questions and database schemas, and its various parameters are also initialized by the encoder parameters of T5. The relation information encoding block uses the relation graph attention network to process data containing relation information. The output of the semantic encoding block is used as input to the relation information encoding block, and the final representation information is the context representation information with relation information, which is used as the output of the encoder. Among them, the focus of the above steps is to complete the construction of the relation graph by mining the relation information between NL question entities and database table / column names. The nodes of the relation graph include NL question entity nodes and database schema nodes (including database table name nodes and database column name nodes), and the edges of the relation graph are the relationships between these nodes. Direct access to the database schema can obtain the relationship between database schema nodes, the relationship between natural language nodes can be obtained through the syntactic analyzer, and the relationship between database schema nodes and natural language nodes can be obtained through string comparison and other methods. After the construction of the relation graph is completed, the relation graph is integrated into the encoder for encoding. In the relation information encoding block, the relation information is calculated as the bias information of the self-attention matrix. After this encoding block, the context representation information with relation information can be obtained as the output of the encoder.

[0167] Specifically, for the relation-aware encoder in the model architecture, relation information is incorporated into the calculation of the attention matrix, and the relation information is used to guide the generation of the next SQL token based on the representation information obtained from the encoder in the previous stage. Furthermore, through the two attention layers of the decoder, relation information is used so that the model has relation information as a substantial guide when performing actual generation tasks, thereby further improving the model's entity alignment capabilities.

[0168] In the first attention layer of the decoder, this application embodiment uses the representation information output by the encoder as a prefix and connects it with the generated SQL statement as the input part of this layer. The purpose of this design is to use the representation information of the encoder to assist the decoder in better associating natural language questions and database structures when generating SQL tokens, thereby improving the accuracy of generation.

[0169] The second attention layer of the decoder acts as the connection between the encoder and the decoder. Here, the output of the encoder serves as the key (K) and value (V) in the attention mechanism calculation, while the output of the previous decoder module serves as the query item (Q) in the attention mechanism calculation. Through this design, the model can utilize relational information at different levels and promote information interaction between different parts, thereby better guiding the decoder to generate SQL statements.

[0170] In the calculation process of the two attention parts of the decoder, the relationship graph attention network is used to process data containing relationship information. This allows the decoder to use relationship information more effectively, helping the model to more accurately understand the relationship between natural language questions and database structures, thereby improving the relevance and accuracy of generated SQL statements. In the relationship graph attention network, the calculation formula related to the attention mechanism is as follows:

[0171]

[0172]

[0173] Among them, represents the initial vector representation of the i-th node, The initial vector representation of the jth neighbor node of the node. The two nodes are connected by a specific relationship, which is represented by r ij express. It is a mapping function that maps the relationship between the i-th node and its j-th neighbor node into a d-dimensional vector. represents the attention weight of the i-th node to the j-th neighbor node, d z represents the dimension of the input vector, Represents the vector representation of the ith node of the input sequence after the ith node is processed by the self-attention mechanism, It represents the vector representation of the ith node in the attention mechanism of the lth layer. Softmax is a function that converts each real number in the attention weight into a probability value between 0 and 1. LayerNorm is a normalization operation. FFN is a feedforward neural network that includes a fully connected layer and an activation function to perform nonlinear transformation on the vector. and is a trainable parameter.

[0174] In the above-mentioned fine-tuning stage, this application embodiment modeled three types of relationship information existing in the NL problem and the database: the relationship between NL problem entities, the relationship between database tables / columns, and the relationship between NL problem entities and database tables / columns, and proposed an algorithm to fuse the relationship information into the encoder and decoder, thereby enhancing the entity alignment capability of the model.

[0175] In order to implement the model training method of the embodiment of the present application, the embodiment of the present application also provides a model training device, such as Fig. 9 As shown, the device comprises:

[0176] The training unit 901 is used to pre-train the first model for multiple tasks based on the training samples; wherein,

[0177] The training samples are generated based on database query codes and corresponding code comments. The multiple tasks include identifier awareness tasks and bimodal learning tasks of natural language and code. The first model is used to convert NL questions into database query codes.

[0178] In one embodiment, the device further comprises:

[0179] An adjustment unit is used to adjust the pre-trained first model based on a set relationship matrix; wherein,

[0180] The set relationship matrix represents the relationship between NL problem entities and / or databases.

[0181] In one embodiment, the training sample further includes:

[0182] Contextual information of the code comments corresponding to the database query code.

[0183] In one embodiment, the training unit 901 performs pre-training of the first model for the identifier perception task, including:

[0184] Training the first model for a first task, the first task being used to predict mask spans of database query codes in training samples;

[0185] Performing a second task training on the first model that has completed the first task training, where the second task is used to mark whether a token of a database query code in a training sample is an identifier;

[0186] The first model that has completed the second task training is trained for a third task, where the third task is used to predict an identifier masked by a database query code in a training sample.

[0187] In one embodiment, the device further comprises:

[0188] The mask processing unit is used to perform mask processing on all identifiers of the database query code in the training sample using the same token before training the first model that has completed the second task training for the third task.

[0189] In one embodiment, when the training unit 901 trains the first model for the dual-modal learning task of natural language and code, the training includes:

[0190] Training the first model for a fourth task, the fourth task being for converting natural language into code;

[0191] The first model is trained on a fifth task for converting code into natural language.

[0192] In one embodiment, the relationship between NL problem entities and / or databases includes one or more of the following:

[0193] The relationship between NL problem entities;

[0194] The relationship between database tables;

[0195] The relationships between database columns;

[0196] The relationship between database tables and columns;

[0197] NL Question The relationship between entities and database tables and / or database columns.

[0198] In one embodiment, the encoder in the first model includes a semantic encoder and a relational encoder, and the output of the semantic encoder serves as the input of the relational encoder; wherein,

[0199] The relation encoder is used to encode the first relation graph as bias information of the self-attention matrix, and the first relation graph is used to characterize the relationship between each natural language question entity node and the database table name node and the database column name node.

[0200] In one embodiment, the decoder in the first model includes a first attention layer and a second attention layer; wherein,

[0201] The output information of the encoder in the first model is connected to the database query code generated by the decoder as the input of the first attention layer; the output information of the encoder in the first model is used as the key and value used by the second attention layer, and the previous output information of the decoder is used as the query item used by the second attention layer.

[0202] In one embodiment, the device further comprises:

[0203] A calculation unit is used to calculate the attention score in the decoder based on the set relationship matrix.

[0204] In actual application, the above-mentioned units can be implemented by a processor in the model training device.

[0205] It should be noted that the model training device provided in the above embodiment is only illustrated by the division of the above program modules. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the model training device provided in the above embodiment belongs to the same concept as the model training method embodiment. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0206] In order to implement the task execution method of the embodiment of the present application, the embodiment of the present application also provides a task execution device, such as Fig.10 As shown, the device comprises:

[0207] An acquisition unit 1001 is used to acquire NL questions;

[0208] A calling unit 1002 is used to call a second model to process the NL problem and obtain a database query statement corresponding to the NL problem; the second model is trained based on any of the above model training methods;

[0209] The execution unit 1003 is used to execute the database query statement to output the query result corresponding to the NL question.

[0210] In actual application, the above-mentioned units can be implemented by a processor or a communication interface in the task execution device.

[0211] It should be noted that the task execution device provided in the above embodiment is only illustrated by the division of the above program modules. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the task execution device provided in the above embodiment and the task execution method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0212] Based on the hardware implementation of the above program module, the embodiment of the present application also provides a first electronic device, such as Fig.11 As shown, the electronic device 1100 includes:

[0213] The first communication interface 1101 is capable of exchanging information with other network nodes;

[0214] The first processor 1102 is connected to the first communication interface 1101 to implement information exchange with other network nodes, and is used to execute the method provided by one or more technical solutions when running a computer program. The computer program is stored in the first memory 1103.

[0215] When the first processor 1102 runs the computer program to execute the above one or more model training methods, specifically:

[0216] The first processor 1102 is configured to:

[0217] Based on the training samples, the first model is pre-trained on multiple tasks; wherein,

[0218] The training samples are generated based on database query codes and corresponding code comments. The multiple tasks include identifier awareness tasks and bimodal learning tasks of natural language and code. The first model is used to convert NL questions into database query codes.

[0219] In one embodiment, the first processor 1102 is further configured to:

[0220] Based on the set relationship matrix, the first model completed by pre-training is adjusted; wherein,

[0221] The set relationship matrix represents the relationship between NL problem entities and / or databases.

[0222] In one embodiment, the training sample based on further includes:

[0223] Contextual information of the code comments corresponding to the database query code.

[0224] In one embodiment, the first processor 1102 pre-trains the first model for an identifier perception task, including:

[0225] Training the first model for a first task, the first task being used to predict mask spans of database query codes in training samples;

[0226] Performing a second task training on the first model that has completed the first task training, where the second task is used to mark whether a token of a database query code in a training sample is an identifier;

[0227] The first model that has completed the second task training is trained for a third task, where the third task is used to predict an identifier masked by a database query code in a training sample.

[0228] In one embodiment, the first processor 1102 is further configured to:

[0229] Before training the first model that has completed the training for the second task for the third task, all identifiers of the database query code in the training sample are masked using the same token.

[0230] In one embodiment, when the first processor 1102 performs a dual-modal learning task of natural language and code on the first model, the training includes:

[0231] Training the first model for a fourth task, the fourth task being for converting natural language into code;

[0232] The first model is trained on a fifth task for converting code into natural language.

[0233] In one embodiment, the relationship between NL problem entities and / or databases includes one or more of the following:

[0234] The relationship between NL problem entities;

[0235] The relationship between database tables;

[0236] The relationships between database columns;

[0237] The relationship between database tables and columns;

[0238] NL Question The relationship between entities and database tables and / or database columns.

[0239] In one embodiment, the encoder in the first model includes a semantic encoder and a relational encoder, and the output of the semantic encoder serves as the input of the relational encoder; wherein,

[0240] The relation encoder is used to encode the first relation graph as bias information of the self-attention matrix, and the first relation graph is used to characterize the relationship between each natural language question entity node and the database table name node and the database column name node.

[0241] In one embodiment, the decoder in the first model includes a first attention layer and a second attention layer; wherein,

[0242] The output information of the encoder in the first model is connected to the database query code generated by the decoder as the input of the first attention layer; the output information of the encoder in the first model is used as the key and value used by the second attention layer, and the previous output information of the decoder is used as the query item used by the second attention layer.

[0243] In one embodiment, the first processor 1102 is further configured to:

[0244] Based on the set relationship matrix, the attention score is calculated in the decoder.

[0245] It should be noted that the specific processing process of the first processor 1102 and the first communication interface 1101 can be understood by referring to the above method.

[0246] Of course, in actual application, the various components in the first electronic device 1100 are coupled together through the first bus system 1104. It can be understood that the first bus system 1104 is used to realize the connection and communication between these components. In addition to the data bus, the first bus system 1104 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig.11 In the figure, various buses are labeled as a first bus system 1104 .

[0247] The first memory 1103 in the embodiment of the present application is used to store various types of data to support the operation of the first electronic device 1100. Examples of such data include: any computer program used to operate on the first electronic device 1100.

[0248] The method disclosed in the above embodiment of the present application can be applied to the first processor 1102, or implemented by the first processor 1102. The first processor 1102 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the first processor 1102 or an instruction in the form of software. The above-mentioned first processor 1102 may be a general-purpose processor, a digital signal processor (DSP, DigitalSignal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The first processor 1102 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in the first memory 1103, and the first processor 1102 reads the information in the first memory 1103, and completes the steps of the above method in combination with its hardware.

[0249] In an exemplary embodiment, the first electronic device 1100 can be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), field programmable gate array (FPGA), general processor, controller, microcontroller (MCU), microprocessor, or other electronic components to execute the aforementioned method.

[0250] Based on the hardware implementation of the above program module, the embodiment of the present application also provides a second electronic device, such as Fig.12 As shown, the electronic device 1200 includes:

[0251] The second communication interface 1201 is capable of exchanging information with other network nodes;

[0252] The second processor 1202 is connected to the second communication interface 1201 to implement information exchange with other network nodes, and is used to execute the method provided by one or more technical solutions when running a computer program. The computer program is stored in the second memory 1203.

[0253] When the second processor 1202 runs the computer program to execute the above one or more task execution methods, specifically:

[0254] The second communication interface 1201 is used for:

[0255] Get NL questions.

[0256] The second processor 1202 is configured to:

[0257] Call the second model to process the NL problem to obtain a database query statement corresponding to the NL problem; the second model is trained based on any of the above-mentioned model training methods; execute the database query statement to output the query result corresponding to the NL problem.

[0258] It should be noted that the specific processing process of the second processor 1202 and the second communication interface 1201 can be understood by referring to the above method.

[0259] Of course, in actual application, the various components in the second electronic device are coupled together through the bus system 1204. It can be understood that the bus system 1204 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1204 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig.12 Various buses are labeled as bus system 1204.

[0260] The second memory 1203 in the embodiment of the present application is used to store various types of data to support the operation of the second electronic device. Examples of such data include: any computer program used to operate on the second electronic device.

[0261] The method disclosed in the above embodiment of the present application can be applied to the second processor 1202, or implemented by the second processor 1202. The second processor 1202 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the second processor 1202. The above-mentioned second processor 1202 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The second processor 1202 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in the second memory 1203, and the second processor 1202 reads the information in the second memory 1203 and completes the steps of the above method in combination with its hardware.

[0262] In an exemplary embodiment, the second electronic device may be implemented by one or more ASICs, DSPs, PLDs, CPLDs, FPGAs, general purpose processors, controllers, MCUs, Microprocessors, or other electronic components to perform the aforementioned method.

[0263] It can be understood that the memory (first memory 1103, second memory 1203) of the embodiment of the present application can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory, a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.

[0264] In an exemplary embodiment, the embodiment of the present application further provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, for example, including a first memory 1103 storing a computer program, and the computer program can be executed by the first processor 1102 of the first electronic device 1100 to complete the steps of the model training method. For another example, a second memory 1203 storing a computer program can be executed by the second processor 1202 of the second electronic device 1200 to complete the steps of the task execution method. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.

[0265] Exemplarily, the embodiment of the present application further provides a computer program product, including a computer program, which can be executed by the first processor 1102 of the first electronic device 1100 to complete the steps of the model training method. Alternatively, the computer program can be executed by the second processor 1202 of the second electronic device 1200 to complete the steps of the task execution method.

[0266] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0267] The term "one or more" herein means any combination of at least two of any one or more of a plurality, for example, including at least one of A, B, and C, and may mean any one or more elements selected from the set consisting of A, B, and C.

[0268] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0269] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A model training method, characterized in that: include: Based on the training samples, the first model is pre-trained on multiple tasks; wherein, The training samples are generated based on database query codes and corresponding code annotations, and the multiple tasks include identifier perception tasks and bimodal learning tasks of natural language and code; The first model is used to convert natural language NL questions into database query codes.

2. The method according to claim 1, characterized in that The method further comprises: Based on the set relationship matrix, the first model completed by pre-training is adjusted; wherein, The set relationship matrix represents the relationship between NL problem entities and / or databases.

3. The method according to claim 1, characterized in that The training samples also include: Contextual information of the code comments corresponding to the database query code.

4. The method according to claim 1, characterized in that: Pre-training the first model for the identifier-aware task includes: Training the first model for a first task, wherein the first task is to predict the mask span of the database query code in the training sample; Performing a second task training on the first model that has completed the first task training, where the second task is used to mark whether a token of a database query code in a training sample is an identifier; The first model that has completed the second task training is trained for a third task, where the third task is used to predict an identifier masked by a database query code in a training sample.

5. The method according to claim 4, characterized in that The method further comprises: Before training the first model trained for the second task for the third task, all identifiers of the database query code in the training sample are masked using the same token.

6. The method according to claim 1, characterized in that The first model is trained on a dual-modal learning task of natural language and code, including: training the first model for a fourth task, the fourth task being to convert natural language into code; and, The first model is trained on a fifth task, wherein the fifth task is to convert the code into natural language.

7. The method according to claim 2, characterized in that The relationship between the NL problem entities and / or databases includes one or more of the following: The relationship between NL problem entities; The relationship between database tables; The relationships between database columns; The relationship between database tables and columns; NL Question The relationship between entities and database tables and / or database columns.

8. The method according to claim 2 or 7, characterized in that: The encoder in the first model includes a semantic encoder and a relational encoder, and the output of the semantic encoder serves as the input of the relational encoder; wherein, The relation encoder is used to encode the first relation graph as bias information of the self-attention matrix, and the first relation graph is used to characterize the relationship between each natural language question entity node and the database table name node and the database column name node.

9. The method according to claim 2 or 7, characterized in that: The decoder in the first model includes a first attention layer and a second attention layer; wherein, The output information of the encoder in the first model is connected to the database query code generated by the decoder as the input of the first attention layer; the output information of the encoder in the first model is used as the key and value used by the second attention layer, and the previous output information of the decoder is used as the query item used by the second attention layer.

10. The method according to claim 9, characterized in that The method further comprises: Based on the set relationship matrix, the attention score is calculated in the decoder.

11. A task execution method, characterized in that: include: Get NL questions; Calling the second model to process the NL problem to obtain a database query statement corresponding to the NL problem; The second model is trained based on the model training method according to any one of claims 1 to 10; Execute the database query statement to output the query result corresponding to the NL question.

12. A model training device, characterized in that: include: A training unit is used to pre-train the first model for multiple tasks based on the training samples; wherein, The training samples are generated based on database query codes and corresponding code annotations, and the multiple tasks include identifier perception tasks and bimodal learning tasks of natural language and code; The first model is used to convert NL questions into database query codes.

13. A task execution device, characterized in that: include: An acquisition unit, used to acquire NL questions; A calling unit, configured to call the second model to process the NL problem and obtain a database query statement corresponding to the NL problem; The second model is trained based on the model training method according to any one of claims 1 to 10; An execution unit is used to execute the database query statement to output the query result corresponding to the NL question.

14. A first electronic device, characterized in that: include: A first processor and a first communication interface; wherein, The first processor is used to pre-train the first model for multiple tasks based on the training samples; wherein, The training samples are generated based on database query codes and corresponding code comments, the multiple tasks include identifier awareness tasks and bimodal learning tasks of natural language and code; the first model is used to convert NL questions into database query codes.

15. A second electronic device, characterized in that: include: A second processor and a second communication interface; wherein, The second processor is configured to: Get NL questions; Calling a second model to process the NL problem to obtain a database query statement corresponding to the NL problem; the second model is trained based on the model training method according to any one of claims 1 to 10; Execute the database query statement to output the query result corresponding to the NL question.

16. An electronic device, characterized in that: include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the steps of the model training method described in any one of claims 1 to 10, or executes the steps of the task execution method described in claim 11.

17. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the model training method described in any one of claims 1 to 10, or implements the steps of the task execution method described in claim 11.

18. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the steps of the model training method described in any one of claims 1 to 10, or implements the steps of the task execution method described in claim 11.

Images