Big language model-based training corpus generation method and device, medium and equipment

Abstract

A training corpus generation method and device based on a large language model, a medium and equipment, relating to the fields of large models, agents and artificial intelligence, the method comprising: inputting the obtained first error sample, execution environment and natural language question sample into a second large language model to obtain a first natural language question for data query and a corresponding first structured query statement, and based on the first natural language question and the first structured query statement, obtaining training corpus for training the first large language model. Thus, automatic generation of training corpus can be realized, which on one hand can increase the number of training corpus and reduce the risk of training corpus exhaustion, thereby better meeting the model training demand in the case of increasing model training demand; on the other hand, human participation can be reduced, and the generation efficiency and quality stability of the training corpus can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of large models, agents and artificial intelligence, in particular, to a large language model-based training corpus generation method, device, medium and equipment. BACKGROUND

[0002] With the continuous development of artificial intelligence technology, large language models are increasingly widely used in daily life. For example, in a data query scenario, in order to quickly obtain target data from a database, a large language model can be used to generate a corresponding query statement according to a data query requirement input by a user in natural language form, and then the query statement is used to retrieve data from the database. The large language model can be trained by training corpus.

[0003] However, the amount of corpus in the related art is limited, which is difficult to meet the training needs of large models, and there is a risk of gradual depletion of training corpus. In addition, since the training corpus in the related art is generally obtained based on manual annotation, the generation efficiency of the training corpus is relatively low. Moreover, due to differences in professional level and / or understanding ability of different annotators, the annotation quality is uneven, which makes it difficult to effectively guarantee the overall quality of the training corpus. SUMMARY

[0004] This summary is provided to introduce a selection of concepts, which are further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in limiting the scope of the claimed subject matter.

[0005] In a first aspect, the present disclosure provides a large language model-based training corpus generation method, which comprises:

[0006] obtaining a first error sample, an execution environment and a natural language question example for data query, wherein the first error sample comprises a sample natural language question for data query, an error structured query statement generated by a first large language model based on the sample natural language question, and a correct structured query statement corresponding to the error structured query statement, and the execution environment is used to represent a query data table obtained based on the first error sample;

[0007] input the first error sample, the execution environment and the natural language question sample into a second large language model to obtain a first natural language question for data query and a first structured query statement corresponding to the first natural language question, wherein the second large language model is used to understand the first error sample to obtain a first query keyword that causes the first large language model to generate the error structured query statement, rewrite the natural language question sample according to the first query keyword and the execution environment to output the first natural language question, and output the first structured query statement according to at least the first natural language question and the execution environment;

[0008] obtain training corpus for training the first large language model based on the first natural language question and the first structured query statement.

[0009] In a second aspect, the present disclosure provides a large language model-based training corpus generation device, which comprises:

[0010] The acquisition module is configured to acquire a first error sample, an execution environment and a natural language question sample for data query, wherein the first error sample comprises a sample natural language question for data query, an error structured query statement generated by a first large language model based on the sample natural language question, and a correct structured query statement corresponding to the error structured query statement, and the execution environment is used to represent a query data table obtained based on the first error sample.

[0011] The first processing module is configured to input the first error sample, the execution environment and the natural language question sample into a second large language model to obtain a first natural language question for data query and a first structured query statement corresponding to the first natural language question, wherein the second large language model is used to understand the first error sample to obtain a first query keyword that causes the first large language model to generate the error structured query statement, rewrite the natural language question sample according to the first query keyword and the execution environment to output the first natural language question, and output the first structured query statement according to at least the first natural language question and the execution environment.

[0012] The second processing module is configured to obtain training corpus for training the first large language model based on the first natural language question and the first structured query statement.

[0013] In a third aspect, the present disclosure provides a computer readable medium having a computer program stored thereon, wherein the computer program is executed by a processing device to implement the steps of the method in the first aspect.

[0014] In a fourth aspect, the present disclosure provides an electronic device, comprising:

[0015] a storage device having stored thereon a computer program;

[0016] a processing device configured to execute the computer program in the storage device to implement the steps of the method in the first aspect.

[0017] In a fifth aspect, the present disclosure provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method in the first aspect.

[0018] By the above technical solution, the obtained first error sample, execution environment and natural language question sample for data query are input into the second large language model to obtain a first natural language question for data query and a first structured query statement corresponding to the first natural language question, and based on the first natural language question and the first structured query statement, a training corpus for training the first large language model is obtained. Since the training corpus is expanded based on the generation of the large language model in the generation process of the training corpus, the automatic generation of the training corpus can be realized, which on the one hand can increase the number of training corpora and reduce the risk of training corpus exhaustion, thereby better meeting the model training demand in the case of increasing model training demand; on the other hand, human intervention can be reduced and the generation efficiency of the training corpus can be improved. In addition, in the generation process of the training corpus, the natural language question sample is rewritten based on the error sample, so that the error information in the training corpus can cover the key error characteristics in the error sample, reduce the interference and redundancy of the remaining noise data, and thus improve the quality stability of the training corpus to a certain extent.

[0019] Other features and advantages of the present disclosure will be described in detail in the following detailed description section. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings in which:

[0021] Figure 1 is a flowchart of a large language model-based training corpus generation method according to an exemplary embodiment of the present disclosure;

[0022] Figure 2 is a flowchart of a large language model-based training corpus generation method according to an exemplary embodiment of the present disclosure;

[0023] Figure 3 This is a schematic diagram of a process for generating a first structured query statement by using a second language model according to an exemplary embodiment of the present disclosure;

[0024] Figure 4 is a flowchart of another method for generating training corpus based on a large language model according to an exemplary embodiment of the present disclosure;

[0025] Figure 5 is a structural block diagram of a training corpus generation device based on a large language model according to an exemplary embodiment of the present disclosure;

[0026] Figure 6 The figure is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0028] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0029] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0030] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0031] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0032] Names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0033] It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the type, use range, use scenario, etc. of personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained in a proper manner according to relevant laws and regulations.

[0034] For example, in response to receiving an active request of a user, prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using personal information of the user. Thus, the user can voluntarily choose whether to provide personal information to the software or hardware such as an electronic device, an application program, a server or a storage medium, etc. that performs the operation of the technical solutions of the present disclosure according to the prompt information.

[0035] As an optional but non-limiting implementation manner, in response to receiving an active request of a user, the manner of sending prompt information to the user may, for example, be a pop-up window manner, in which the prompt information can be presented in the form of text. In addition, the pop-up window can also carry selection controls for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0036] It can be understood that the above notification and user authorization process is only illustrative and does not limit the implementation manners of the present disclosure, and other manners that meet relevant laws and regulations can also be applied to the implementation manners of the present disclosure.

[0037] Meanwhile, it can be understood that the data (including but not limited to the data itself, the acquisition or use of the data) involved in the present technical solutions should comply with the requirements of relevant laws and regulations and related provisions.

[0038] In a data query scenario, in order to quickly obtain target data from a database, a large language model can usually be used to generate a corresponding query statement according to a data query requirement input by a user in a natural language form, and then the query statement is used to retrieve data from the database. For example, an advertiser needs to query the delivery data in order to timely understand the advertising delivery effect. In this scenario, the advertiser can query the delivery data in a natural language interactive database query product based on a large language model. The large language model can be obtained by training a corpus.

[0039] However, in the related art, the training scheme of the large language model depends on screening data from the Internet and low-density open corpus in the domain for training. However, as the model training demand continues to rise, the existing corpus resources have gradually failed to meet the higher requirements of model training. Specifically, there are the following problems:

[0040] Training data exhaustion: As the model size of large models continues to expand, the parameter size typically grows exponentially, and since each parameter of the model needs sufficient data to adjust its weight to ensure that the model can accurately learn and generalize. However, the limited inventory of the related technology corpus has been difficult to meet the training needs of larger size models, leading to an increasingly prominent risk of data exhaustion;

[0041] Low yield and low efficiency of manual annotation: As the model size of large models continues to expand, the demand for annotated data also grows exponentially. However, the yield of manual annotation is limited, making it difficult to meet the needs of large-scale model training. In addition, in the process of manual annotation, annotators need to analyze, understand and annotate data one by one, which not only consumes a lot of time, but also increases the difficulty of annotation when facing complex data or highly professional content, resulting in relatively low efficiency of training corpus generation;

[0042] Training quality is difficult to guarantee: Internet public corpus, domain corpus and manually annotated corpus may contain a large amount of noise data, such as syntax errors, spelling errors and irrelevant content, etc., resulting in uneven quality of training corpus, making it difficult to ensure the quality stability of the training process.

[0043] Therefore, the present disclosure provides a large language model-based training corpus generation method, device, medium and equipment to solve the above technical problems.

[0044] The embodiments of the present disclosure are further explained and described below in conjunction with the accompanying drawings.

[0045] Figure 1 is a flowchart of a large language model-based training corpus generation method according to an exemplary embodiment of the present disclosure, referring to Figure 1 The large language model-based training corpus generation method can include the following steps:

[0046] S101: obtaining a first error sample, an execution environment and a natural language question sample for data query, wherein the first error sample includes a sample natural language question for data query, an error structured query statement generated by a first large language model based on the sample natural language question, and a correct structured query statement corresponding to the error structured query statement, and the execution environment is used to represent the query data table obtained based on the first error sample.

[0047] In this embodiment, the query keywords in the first error sample can be extracted by the execution environment. For example, if the first error sample includes the sample natural language question "overall yield rate", the error structured query statement "avg(`income` / `revenue`) | sum(`income` / `revenue`)", and the correct structured query statement "sum(`income`) / sum(`revenue`)", the query keywords in the first error sample can be extracted to obtain the execution environment "`income`, int, \n `revenue`, int".

[0048] For example, if the first error sample includes the sample natural language question "average yield rate (income / revenue)", the error structured query statement "sum(`income`) / sum(`revenue`) | sum(`income` / `revenue`)", and the correct structured query statement "avg(`income` / `revenue`)", the query keywords in the first error sample can be extracted to obtain the execution environment "`income`, int, \n `revenue`, int".

[0049] It should be understood that if the first error sample includes multiple, when determining the execution environment, the error type of each first error sample can be determined first, then clustered based on the error type to obtain error sets of different error types, and for each error set, the query keywords in each first error sample under the error set can be extracted to obtain the execution environment corresponding to different error types.

[0050] In this embodiment, the first error sample can be obtained by the following method:

[0051] obtaining an error definition for the first large language model, a sample natural language question for data query, and a third structured query statement generated by the first large language model based on the sample natural language question, wherein the error definition includes processing links required by the first large language model to generate a structured query statement, error types under each processing link, and error details corresponding to each error type; inputting the sample natural language question, the third structured query statement and the error definition into the second large language model to obtain the first error sample, wherein the second large language model is further used to understand the sample natural language question and the third structured query statement based on the error definition to obtain an error structured query statement, generate a correct structured query statement based on the error structured query statement and the sample natural language question corresponding to the error structured query statement, and output the first error sample based on the error structured query statement, the sample question corresponding to the error structured query statement, and the correct structured query statement.

[0052] It should be understood that when generating structured query statements based on a large language model, the required processing steps generally include two steps: question understanding and logical processing. Therefore, the processing steps in this embodiment can include question understanding and logical processing. The question understanding step is used to identify user intent based on the input natural language question, and the logical processing step is used to perform logical reasoning based on the identified user intent. Through a large number of experiments, it was found that the error types corresponding to the question understanding step and the logical processing step, as well as the error details corresponding to each error type, can be shown in Table 1.

[0053] Table 1. Error definitions for the top language model

[0054]

[0055] It should be understood that this is merely an illustrative description and does not constitute a limitation on the solution. Limit and sample are commonly used clauses in structured query statements to limit the query result set.

[0056] In addition, it should be understood that when the first language model generates the third structured query statement based on the sample natural language question, the generated third structured query statement may be correct or incorrect. Therefore, in order to obtain a more accurate error sample, after obtaining the third structured query statement corresponding to the sample natural language question through the first language model, the sample natural language question, the third structured query statement and the error definition can be input into the second language model, so that the second language model can identify whether the third structured query statement is correct based on the error definition and the sample natural language question. If the third structured query statement is incorrect, it will be treated as an erroneous structured query statement, and it can be corrected in combination with the corresponding sample natural language question to generate a correct structured query statement, so that the first error sample can be obtained based on the erroneous structured query statement, the sample question corresponding to the erroneous structured query statement and the correct structured query statement.

[0057] S102: Input the first error sample, the execution environment, and the natural language question sample into the second largest language model to obtain a first natural language question for data query and a first structured query statement corresponding to the first natural language question, wherein the second largest language model is used to understand the first error sample, obtain a first query keyword that causes the first largest language model to generate an erroneous structured query statement, rewrite the natural language question sample according to the first query keyword and the execution environment, output the first natural language question, and output the first structured query statement at least based on the first natural language question and the execution environment.

[0058] For example, the first prompt word template and the second prompt word template can be pre-set, where the first prompt word template is used to generate the first natural language question, and the second prompt word template is used to generate the first structured query statement. Thus, after obtaining the first error sample, the execution environment, and the natural language question sample, the first error sample, the execution environment, and the natural language question sample can be filled into the first prompt word template to obtain the first prompt word, and the first prompt word can be input into the second large language model to obtain the first natural language question with the error in the first error sample, as shown in Figure 2 Then, the first natural language question, the execution environment, and the first error sample can be filled into the second prompt word template to obtain the second prompt word, and the second prompt word can be input into the second large language model to obtain the corresponding first structured query statement with the error in the first error sample, as shown in Figure 3 The first prompt word template and the second prompt word template can be determined according to actual conditions, and the embodiments of the present disclosure do not make any limitation on this.

[0059] S103: Obtain training corpus for training the first large language model based on the first natural language question and the first structured query statement.

[0060] Through the above technical solution, the obtained first error sample, execution environment, and natural language question sample for data query can be input into the second large language model to obtain the first natural language question for data query and the first structured query statement corresponding to the first natural language question, and based on the first natural language question and the first structured query statement, the training corpus for training the first large language model can be obtained. Since the training corpus is expanded based on the generation of the large language model in the generation process of the training corpus, the automatic generation of the training corpus can be realized, which can increase the number of training corpora on one hand, reduce the risk of training corpus exhaustion, and thus better meet the model training demand in the case of increasing model training demand; on the other hand, it can reduce human intervention and improve the generation efficiency of the training corpus. In addition, in the generation process of the training corpus, the natural language question sample is rewritten based on the error sample, so that the error information in the training corpus can cover the key error characteristics in the error sample, reduce the interference and redundancy of the remaining noise data, and thus improve the quality stability of the training corpus to a certain extent.

[0061] In addition, in the process of generating the training corpus, the thinking boundary of the second large language model can be defined by the execution environment, so that the generated training corpus can meet the actual business demand and further improve the quality of the training corpus.

[0062] To facilitate understanding of the large language model-based training corpus generation method provided by the present disclosure, the possible implementation manners in the present disclosure are described below.

[0063] In a possible manner, the first error sample can further include an error label, the error label representing an error cause of the error structured query statement, and accordingly, the second large language model is configured to output the first natural language question in the following manner:

[0064] Based on the error label, a target error type of the error structured query statement is determined; and based on the target error type, a second execution environment is determined from a plurality of preset first execution environments, each of the first execution environments being previously associated with an error type of the error structured query statement generated by the first large language model, and the second execution environment being associated with the same error type as the target error type; and based on the first query keyword and the second execution environment, the natural language question sample is rewritten to output the first natural language question.

[0065] For example, a plurality of error labels can be preset, and each of the error labels includes at least an error type. Thus, after obtaining the error structured query statement corresponding to the sample natural language question, the error structured query statement can be analyzed in combination with the plurality of preset error labels and the sample natural language question to obtain an error label for the error structured query statement, or after obtaining the error structured query statement corresponding to the sample natural language question, the sample natural language question, the error structured query statement, and the plurality of preset error labels can be input into a large model to output an error label for the error structured query statement by the large model.

[0066] It should be understood that the content contained in the error label can be determined according to actual conditions, and the embodiments of the present disclosure do not make any limitation thereon. For example, in order to accurately indicate the error cause of the error structured query statement, the content contained in the error label can include a processing link required by the large language model to generate the structured query statement, a corresponding error type under the processing link, and a corresponding error detail point under the error type. For example, if the first large language model has a problem in the mean calculation process under the question understanding link in the process of generating the structured query statement based on the sample natural language question, the error label can be: question understanding-computation method-mean calculation.

[0067] It should be understood that different error types generally correspond to different execution environments, and thus an execution environment can be constructed for each error type in advance. For example, the plurality of preset first execution environments can be obtained in the following manner:

[0068] A plurality of different first error samples are obtained, and for each first error sample, the error type of the erroneous structured query statement in the first error sample is determined based on the error label included in the first error sample; field identification is performed on the first error samples with the same error type to obtain an initial data field corresponding to the error type, and the initial data field is de-duplicated to obtain a de-duplicated data field corresponding to the error type; data table construction is performed based on the de-duplicated data field corresponding to the error type and the field type corresponding to the de-duplicated data field to obtain a first execution environment corresponding to the error type.

[0069] After obtaining a plurality of preset first execution environments and a target error type, error type matching can be performed according to the target error type and the error types associated with the first execution environments to obtain a second execution environment with the same error type as the target error type. After obtaining the second execution environment, the natural language question sample can be rewritten based on the first query keyword and the second execution environment to obtain a first natural language question with the error in the first error sample.

[0070] In the above manner, a plurality of first execution environments can be constructed in advance, and different first execution environments can correspond to different error types. Thus, the corresponding first execution environment can be selected for subsequent processing according to the error type of the erroneous structured query statement. Compared with constructing a general execution environment including different error types, on the one hand, the data processing amount of the second large language model can be reduced, and the data processing efficiency can be improved; on the other hand, since the selected execution environment is more matched with the current natural language question sample, the generated first natural language question can meet the actual business requirements in the process of rewriting the natural language question sample based on the first query keyword and the second execution environment, i.e., the data query can be performed in the second execution environment, thereby further improving the quality of the training corpus.

[0071] In a possible manner, the second large language model can output the first structured query statement in the following manner:

[0072] The sample natural language question and the erroneous structured query statement in the first error sample are understood to obtain first knowledge for clarifying the erroneous structured query statement, and the sample natural language question and the correct structured query statement in the first error sample are understood to obtain second knowledge for generating the correct structured query statement; the first structured query statement corresponding to the first natural language question is generated according to the first knowledge, the second knowledge, the first natural language question and the execution environment; and the first structured query statement is output.

[0073] It should be understood that the second large language model can make the knowledge of the incorrect structured query statement explicit by understanding the sample natural language question and the incorrect structured query statement in the first error sample. At the same time, the second large language model can make the knowledge of generating the correct structured query statement explicit by understanding the sample natural language question and the correct structured query statement in the first error sample. Thus, when the second large language model generates the first structured query statement corresponding to the first natural language question according to the first knowledge, the second knowledge, the first natural language question and the execution environment, the error in the first structured query statement can be identified in view of the correct knowledge and the existing error knowledge, thereby further improving the diversity and richness of the training corpus.

[0074] In addition, it should be understood that in the related art, the training quality is generally improved by model distillation. However, due to the lack of targeted training for specific problems and error correction, the problem of lack of pertinence exists. In the embodiment, by using the first knowledge and the second knowledge, the directional generation of the structured query statement for the known problem is performed, so that the generated training corpus can be more in line with the actual demand, thereby helping the large language model to better learn the characteristics and rules of specific tasks when the large language model is trained based on the training corpus, and improving the training effect of the large language model.

[0075] In a possible manner, based on the first natural language question and the first structured query statement, the training corpus for training the first large language model can include:

[0076] combining the first natural language question and the first structured query statement into a first training corpus; training the first large language model based on the first training corpus to obtain a third structured query statement generated by the first large language model based on the first natural language question in the training process; in a case where the third structured query statement is an incorrect structured query statement, obtaining a second error sample based on the first natural language question, the first structured query statement and the third structured query statement; inputting the second error sample, the execution environment and the natural language question sample into the second large language model to obtain a second natural language question for data query output by the second large language model and a second structured query statement corresponding to the second natural language question; and obtaining a second training corpus for training the first large language model based on the second natural language question and the second structured query statement.

[0077] It should be understood that the third structured query statement generated by the first large language model based on the first natural language question during the training process may be correct or incorrect. Therefore, in order to further improve the diversity and richness of the training corpus, when the generated third structured query statement is incorrect, a new error sample, that is, a second error sample, can be constructed based on the first natural language question, the first structured query statement and the third structured query statement. Then, the execution environment of the second error sample and the natural language question example are input into the second large language model to obtain a new second training corpus for training the first large language model. If there are still errors in the output structured query statement during the training of the large model based on the second training corpus, the above steps can be re-executed, such as Figure 4 shown.

[0078] This approach creates a closed-loop production loop for training data, providing an effective solution for expanding it. Specifically, this closed-loop production process allows for continuous production of training data to address any issues that arise during large model training. This allows for targeted optimization of large model performance, gradually improving its accuracy and reliability when generating structured queries for natural language problems.

[0079] In a possible manner, obtaining training corpus for training the first language model based on the first natural language question and the first structured query statement may include:

[0080] A syntax check is performed on the first structured query statement to obtain a syntax check result, and based on the syntax check result, a grammatically compliant structured query statement is screened from the first structured query statement; a spot check is performed on the compliant structured query statement to obtain a spot-checked structured query statement, and a matching result between the spot-checked structured query statement and the corresponding first natural language question is determined, wherein the matching result is used to characterize whether the data required for the corresponding first natural language question can be queried through the spot-checked structured query statement; when the matching result meets the preset conditions, a compliant structured query statement and the corresponding first natural language question are combined into a training corpus for training the first large language model.

[0081] For example, if the first structured query statement includes multiple statements, a syntax check can be performed on each first structured query statement. If the syntax check result indicates that the first structured query statement has a syntax problem, the first natural language question and the first structured query statement can be discarded. Alternatively, the first structured query statement can be modified and the modified first structured query statement can be used as a compliant structured query statement. If the syntax check result indicates that the first structured query statement does not have a syntax problem, it can be used as a compliant structured query statement. After obtaining all compliant structured query statements, 10% of the compliant structured query statements can be extracted as spot-checked structured query statements, and for each spot-checked structured query statement, it can be determined whether the spot-checked structured query statement can query the data required by the corresponding first natural language question. If the sampled structured query statement can query the data required for the corresponding first natural language question, then the sampled structured query statement and the corresponding first natural language question are combined into a training corpus for training the first large language model; if the sampled structured query statement cannot query the data required for the corresponding first natural language question, then the sampled structured query statement and the corresponding first natural language question can be discarded, or the sampled structured query statement can be modified accordingly, and then the modified sampled structured query statement and the corresponding first natural language question are combined into a training corpus for training the first large language model.

[0082] Through the above method, a compliant structured query statement can be obtained by grammatically checking the first structured query statement, and a compliant structured query statement can be obtained by spot-checking the compliant structured query statement to obtain a compliant structured query statement that can query the data required for the corresponding first natural language question, thereby obtaining a training corpus for training the first large language model. This can reduce errors in the training corpus and improve the quality of the training corpus. Furthermore, when the large model is trained based on this training corpus, the training effect of the large model can be improved.

[0083] Based on the same concept, the embodiment of the present disclosure also provides a training corpus generation device based on a large language model, such as Figure 5 As shown, the training corpus generation device 500 based on the large language model may include:

[0084] Acquisition module 501 is configured to acquire a first error sample, an execution environment, and a sample natural language question for data query, wherein the first error sample includes a sample natural language question for data query, an erroneous structured query statement generated by the first language model based on the sample natural language question, and a correct structured query statement corresponding to the erroneous structured query statement, and the execution environment is configured to represent a query data table obtained based on the first error sample;

[0085] The first processing module 502 is configured to input the first error sample, the execution environment, and the natural language question sample into a second large language model to obtain a first natural language question for data query and a first structured query statement corresponding to the first natural language question, wherein the second large language model is configured to understand the first error sample to obtain a first query keyword that causes the first large language model to generate an error structured query statement, rewrite the natural language question sample according to the first query keyword and the execution environment, and output the first natural language question, and output the first structured query statement according to at least the first natural language question and the execution environment.

[0086] The second processing module 503 is configured to obtain training corpus for training the first large language model based on the first natural language question and the first structured query statement.

[0087] The training corpus generation apparatus 500 based on the large language model can input the obtained first error sample, execution environment, and natural language question sample for data query into the second large language model to obtain a first natural language question for data query and a first structured query statement corresponding to the first natural language question, and obtain training corpus for training the first large language model based on the first natural language question and the first structured query statement. Since the training corpus is expanded based on the generation of the large language model during the generation of the training corpus, the automatic generation of the training corpus can be realized, which can increase the quantity of the training corpus and reduce the risk of exhaustion of the training corpus, thereby better meeting the model training demand in the case of increasing model training demand, and can reduce human intervention and improve the generation efficiency of the training corpus. In addition, since the natural language question sample is rewritten based on the error sample during the generation of the training corpus, the error information in the training corpus can be reduced in the interference and redundancy of the remaining noise data while covering the key error characteristics of the error sample, thereby improving the quality stability of the training corpus to a certain extent.

[0088] In addition, since the thinking boundary of the second large language model can be defined by the execution environment during the generation of the training corpus, the generated training corpus can meet the actual business demand and further improve the quality of the training corpus.

[0089] In a possible manner, the first error sample further includes an error label, and the error label represents an error reason of the error structured query statement. Accordingly, the first processing module 502 can include:

[0090] The determining sub-module is configured to determine the second execution environment from a plurality of preset first execution environments according to a target error type, wherein each first execution environment is associated with an error type of an error structured query statement generated by a first large language model, and the second execution environment is associated with the same error type as the target error type.

[0091] The first processing sub-module is configured to rewrite the natural language problem sample according to the first query keyword and the second execution environment, and output a first natural language problem.

[0092] In a possible manner, the plurality of preset first execution environments are obtained in the following manner:

[0093] A plurality of different first error samples are obtained, and for each first error sample, an error type of an error structured query statement in the first error sample is determined based on an error label included in the first error sample.

[0094] Field recognition is performed on the first error samples with the same error type to obtain an initial data field corresponding to the error type, and the initial data field is subjected to a de-duplication process to obtain a de-duplicated data field corresponding to the error type.

[0095] A data table is constructed based on the de-duplicated data field corresponding to the error type and a field type corresponding to the de-duplicated data field to obtain a first execution environment corresponding to the error type.

[0096] In a possible manner, the first processing module 502 can include:

[0097] The understanding sub-module is configured to understand a sample natural language problem and an error structured query statement in the first error sample to obtain first knowledge for clarifying the error structured query statement, and understand a sample natural language problem and a correct structured query statement in the first error sample to obtain second knowledge for generating the correct structured query statement.

[0098] The generating sub-module is configured to generate a first structured query statement corresponding to the first natural language problem according to the first knowledge, the second knowledge, the first natural language problem, and the execution environment.

[0099] The output sub-module is configured to output the first structured query statement.

[0100] In a possible manner, the second processing module 503 can include:

[0101] The second processing sub-module is configured to combine the first natural language problem and the first structured query statement into a first training corpus.

[0102] The training submodule is configured to train the first large language model based on a first training corpus, to obtain a third structured query statement generated by the first large language model based on a first natural language question in a training process;

[0103] The third processing submodule is configured to, in a case where the third structured query statement is an incorrect structured query statement, obtain a second incorrect sample based on the first natural language question, the first structured query statement, and the third structured query statement;

[0104] The fourth processing submodule is configured to input the second incorrect sample, an execution environment, and a natural language question sample into a second large language model, to obtain a second natural language question for data query output by the second large language model and a second structured query statement corresponding to the second natural language question;

[0105] The fifth processing submodule is configured to obtain a second training corpus for training the first large language model based on the second natural language question and the second structured query statement.

[0106] In a possible manner, the first incorrect sample is obtained in the following manner:

[0107] An error definition for the first large language model, a sample natural language question for data query, and a third structured query statement generated by the first large language model based on the sample natural language question are obtained, wherein the error definition includes a processing link required by the first large language model to generate a structured query statement, an error type under each processing link, and an error detail point corresponding to each error type;

[0108] The sample natural language question, the third structured query statement, and the error definition are input into a second large language model to obtain the first incorrect sample, wherein the second large language model is further configured to understand the sample natural language question and the third structured query statement based on the error definition, to obtain an incorrect structured query statement, generate a correct structured query statement based on the incorrect structured query statement and the sample natural language question corresponding to the incorrect structured query statement, and output the first incorrect sample based on the incorrect structured query statement, the sample question corresponding to the incorrect structured query statement, and the correct structured query statement.

[0109] In a possible manner, the second processing module 503 can include:

[0110] The checking submodule is configured to perform syntax checking on the first structured query statement to obtain a syntax checking result, and filter a compliant structured query statement that is syntax-compliant from the first structured query statement according to the syntax checking result;

[0111] The sixth processing sub-module is configured to perform spot-checking on the compliance structured query statement, to obtain a spot-checked structured query statement, and to determine a matching result between the spot-checked structured query statement and the corresponding first natural language question, wherein the matching result is used to represent whether the corresponding first natural language question can be queried by using the spot-checked structured query statement.

[0112] The combination sub-module is configured to combine a compliance structured query statement and a corresponding first natural language question into a training corpus for training the first large language model, in a case where the matching result satisfies a preset condition.

[0113] Reference will be made to the following description Figure 6 , which shows a structural diagram of an electronic device 600 suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Personal Computers), PMPs (Portable Multimedia Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, as well as fixed terminals such as digital TVs, desktop computers, and the like. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0114] As shown in Figure 6 , the electronic device 600 can include a processing device (such as a central processor, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or loaded into a random access memory (RAM) 603 from a storage device 608. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0115] Generally, the following devices can be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 608 including, for example, a magnetic tape, a hard disk, and the like; and communication devices 609. The communication devices 609 can allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that all of the shown devices are not required to be implemented or possessed. More or fewer devices can be alternatively implemented or possessed.

[0116] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program in accordance with embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program comprising program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0117] It should be noted that the computer-readable medium described above in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the computer-readable program code is carried. Such a propagated data signal can take a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that can be used to carry or store program code for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wire, cable, optical fiber, RF, etc., or any suitable combination thereof.

[0118] In some embodiments, any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol) can be utilized for communication, and can be interconnected with digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

[0119] The computer readable medium described above can be included in the electronic device described above; or can exist separately, without being assembled into the electronic device.

[0120] The computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to: obtain a first error sample, an execution environment, and a natural language question sample for a data query, wherein the first error sample includes a sample natural language question for a data query, an error structured query statement generated by a first large language model based on the sample natural language question, and a correct structured query statement corresponding to the error structured query statement, and the execution environment is used to represent a query data table obtained based on the first error sample; input the first error sample, the execution environment, and the natural language question sample into a second large language model to obtain a first natural language question for the data query and a first structured query statement corresponding to the first natural language question, wherein the second large language model is used to understand the first error sample to obtain a first query keyword that causes the first large language model to generate the error structured query statement, rewrite the natural language question sample according to the first query keyword and the execution environment to output the first natural language question, and output the first structured query statement according to at least the first natural language question and the execution environment; and obtain training corpus for training the first large language model based on the first natural language question and the first structured query statement.

[0121] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0122] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0123] The modules involved in the embodiments of the present disclosure can be implemented in the manner of software or hardware. Among them, the name of the module does not constitute a limitation to the module itself in some cases.

[0124] The functions described in the foregoing description can be implemented, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0125] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0126] The above description is only preferred embodiments of the present disclosure and a description of the principles of the technology used. It should be understood by those skilled in the art that the scope of the disclosure involved in the disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and also covers other technical solutions formed by any combination of the above technical features or equivalent features without departing from the above disclosed concept. For example, the technical solutions formed by replacing the above features with the technical features disclosed in the disclosure (but not limited to) having similar functions.

[0127] In addition, although each operation is described in a particular order, this should not be understood as requiring the operations to be performed in the specific order shown or in a sequential order. In certain circumstances, multitasking and parallel processing can be advantageous. Similarly, although several implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the disclosure. Certain features described in the context of separate embodiments can also be combined in a single embodiment. Conversely, various features described in the context of a single embodiment can also be separated and implemented in multiple embodiments.

[0128] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. With respect to the devices in the above-described embodiments, in which various modules perform operations, the specific manner in which the operations are performed by the various modules has been described in detail in the embodiments related to the method, and will not be described in detail here.

Claims

1. A method for generating a training corpus based on a large language model, characterized in that, The method for generating training corpus based on a large language model comprises the following steps: obtaining a first error sample, an execution environment, and a natural language question sample for data query, wherein the first error sample comprises a sample natural language question for data query, an error structured query statement generated by a first large language model based on the sample natural language question, and a correct structured query statement corresponding to the error structured query statement, and the execution environment is used to represent a query data table obtained based on the first error sample; inputting the first error sample, the execution environment, and the natural language question sample into a second large language model to obtain a first natural language question for data query and a first structured query statement corresponding to the first natural language question, wherein the second large language model is used to understand the first error sample to obtain a first query keyword causing the first large language model to generate the error structured query statement, rewrite the natural language question sample according to the first query keyword and the execution environment to output the first natural language question, and output the first structured query statement according to at least the first natural language question and the execution environment; based on the first natural language question and the first structured query statement, obtaining training corpus for training the first large language model.

2. The method of claim 1, wherein the method further comprises: The first error sample further comprises an error label representing the error reason of the error structured query statement, and the second large language model is used to output the first natural language question in the following manner: determining a target error type of the error structured query statement based on the error label; determining a second execution environment from a plurality of preset first execution environments according to the target error type, wherein each of the first execution environments is previously associated with an error type of an error structured query statement generated by the first large language model, and the second execution environment is associated with the same error type as the target error type; rewriting the natural language question sample according to the first query keyword and the second execution environment to output the first natural language question.

3. The method of claim 2, wherein the method further comprises: The plurality of preset first execution environments are obtained in the following manner: obtaining a plurality of different first error samples, and for each of the first error samples, determining an error type of an error structured query statement in the first error sample based on an error label included in the first error sample; performing field identification on the first error samples with the same error type to obtain an initial data field corresponding to the error type, and performing de-duplication processing on the initial data field to obtain a de-duplicated data field corresponding to the error type; performing data table construction based on the de-duplicated data field corresponding to the error type and a field type corresponding to the de-duplicated data field to obtain a first execution environment corresponding to the error type.

4. The method of claim 1-3, wherein, The second large language model is used to output the first structured query statement in the following manner: The sample natural language question and the erroneous structured query statement in the first error sample are understood to obtain first knowledge for clarifying the erroneous structured query statement, and the sample natural language question and the correct structured query statement in the first error sample are understood to obtain second knowledge for generating a correct structured query statement; Generate a first structured query statement corresponding to the first natural language question based on the first knowledge, the second knowledge, the first natural language question, and the execution environment; The first structured query statement is output.

5. The method of claim 4, wherein the method further comprises: The obtaining, based on the first natural language question and the first structured query statement, a training corpus for training the first language model includes: Combining the first natural language question and the first structured query statement into a first training corpus; Training the first language model based on the first training corpus to obtain a third structured query statement generated by the first language model based on the first natural language question during the training process; In a case where the third structured query statement is an erroneous structured query statement, obtaining a second error sample based on the first natural language question, the first structured query statement, and the third structured query statement; Inputting the second error sample, the execution environment, and the natural language question example into the second large language model, obtaining a second natural language question for data query output by the second large language model and a second structured query statement corresponding to the second natural language question; Based on the second natural language question and the second structured query statement, a second training corpus for training the first language model is obtained.

6. The method of claim 1-3, wherein, The first error sample is obtained in the following manner: Obtaining an error definition for the first language model, a sample natural language question for data query, and a third structured query statement generated by the first language model based on the sample natural language question, wherein the error definition includes processing steps required for the first language model to generate the structured query statement, an error type for each processing step, and error details corresponding to each error type; The sample natural language question, the third structured query statement and the error definition are input into the second largest language model to obtain a first error sample, wherein the second largest language model is also used to understand the sample natural language question and the third structured query statement based on the error definition to obtain an erroneous structured query statement, generate a correct structured query statement based on the erroneous structured query statement and the sample natural language question corresponding to the erroneous structured query statement, and output the first error sample based on the erroneous structured query statement, the sample question corresponding to the erroneous structured query statement and the correct structured query statement.

7. The method of claim 1-3, wherein, The obtaining, based on the first natural language question and the first structured query statement, a training corpus for training the first language model includes: The first structured query statement is syntax checked to obtain a syntax checking result, and a syntax-compliant compliant structured query statement is screened from the first structured query statement according to the syntax checking result; The compliant structured query statement is spot-checked to obtain a spot-checked structured query statement, and a matching result between the spot-checked structured query statement and the corresponding first natural language question is determined, wherein the matching result is used to represent whether the corresponding first natural language question required data can be queried by the spot-checked structured query statement; In a case where the matching result meets a preset condition, the compliant structured query statement and the corresponding first natural language question are combined as a training corpus for training the first large language model. 8.A device for generating a training corpus based on a large language model, characterized in that, The training corpus generation device based on the large language model comprises: An acquisition module is configured to acquire a first error sample, an execution environment, and a natural language question sample for data query, wherein the first error sample includes a sample natural language question for data query, an error structured query statement generated by a first large language model based on the sample natural language question, and a correct structured query statement corresponding to the error structured query statement, and the execution environment is used to represent a query data table obtained based on the first error sample; A first processing module is configured to input the first error sample, the execution environment, and the natural language question sample into a second large language model to obtain a first natural language question for data query and a first structured query statement corresponding to the first natural language question, wherein the second large language model is used to understand the first error sample to obtain a first query keyword that causes the first large language model to generate the error structured query statement, rewrite the natural language question sample according to the first query keyword and the execution environment, and output the first natural language question, and output the first structured query statement according to at least the first natural language question and the execution environment; A second processing module is configured to obtain a training corpus for training the first large language model based on the first natural language question and the first structured query statement.

9. A computer readable medium having stored thereon a computer program, characterized in that, The computer program is executed by the processing device to implement the steps of the method of any one of claims 1-7.

10. An electronic device, comprising: Comprises: A storage device having a computer program stored thereon; A processing device configured to execute the computer program in the storage device to implement the steps of the method of any one of claims 1-7.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1-7. The computer program is executed by the processor to implement the steps of the method of any one of claims 1-7.

Images