Method, apparatus, device, medium and product for generating question-answer data

Abstract

Disclosed are methods, devices, equipment, media and products for generating question-and-answer data, and relate to the technical fields of language models and large language models. The method includes selecting a first question from questions extracted from a log based on the clarity of the question. It also includes selecting a second question from the first question based on the question meta-information of the first question, and the number of the second questions is less than or equal to the number of the first questions. It also includes assigning a true answer to the second question based on a true answer item from a database. It also includes generating a model answer for the second question based on a model answer item generated by a language model. It also includes associating the second question with one or more answers whose answer differences in the true answer and the model answer meet a difference threshold and storing them to generate a question-and-answer set based on a comparison between the true answer and the model answer. In this way, a high-quality annotated data set can be automatically generated to ensure that the questions are clear and targeted, and the coverage and accuracy of the answers are guaranteed.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to the field of computers, and in particular, relate to methods and apparatuses, devices, media, and products for generating question and answer data. Background Art

[0002] In recent years, language models have developed rapidly and have become an influential and highly-watched technology. As a representative of language models, large language models (LLMs) are widely used for their powerful natural language understanding and generalization capabilities. One of the core functions of language models is to significantly simplify the previously complex and cumbersome knowledge acquisition process. Users can overcome the high knowledge barriers of the past and achieve functions equivalent to those difficult-to-use professional tools, thereby reducing the entry threshold and learning costs of knowledge.

[0003] The evaluation of language model capabilities and performance is crucial, as it is directly related to the performance of the model in practical applications and is a key criterion for measuring whether it can meet processing requirements and achieve expected results. Through scientific and comprehensive evaluation, problems with language models can be discovered in a timely manner, and targeted optimization measures can be taken to continuously improve the accuracy and reliability of the model to meet the growing processing needs. Summary of the invention

[0004] An embodiment of the present disclosure provides a solution for generating question and answer data.

[0005] In a first aspect of the present disclosure, a method for generating question-and-answer data is provided, the method comprising selecting a first question from questions extracted from a log based on the clarity of the question. The method also comprises selecting a second question from the first question based on question meta-information of the first question, the number of the second questions being less than or equal to the number of the first questions. The method also comprises assigning a true answer to the second question based on a true answer item from a database. The method also comprises generating a model answer for the second question based on a model answer item generated by a language model. The method also comprises, based on a comparison between the true answer and the model answer, associating the second question with one or more answers whose answer differences between the true answer and the model answer meet a difference threshold and storing them to generate a question-and-answer set.

[0006] In a second aspect of the present disclosure, a device for generating question-and-answer data is provided, the device comprising a first selection module configured to select a first question among questions extracted from a log based on the clarity of the question. The device also comprises a second selection module configured to select a second question among the first questions based on the question meta-information of the first question, the number of the second questions being less than or equal to the number of the first questions. The device also comprises an answer assignment module configured to assign a true answer to the second question based on a true answer item from a database. The device also comprises an answer generation module configured to generate a model answer for the second question based on a model answer item generated by a language model. The device also comprises a question set generation module configured to associate the second question with one or more answers whose answer differences between the true answer and the model answer meet a difference threshold and store them to generate a question-and-answer set based on a comparison between the true answer and the model answer.

[0007] According to a third aspect of the present disclosure, an electronic device is provided. The computing device includes a processor and a memory, wherein instructions are stored in the memory, and when the instructions are executed by the processor, the processor executes a method or process according to an embodiment of the present disclosure.

[0008] According to a fourth aspect of the present disclosure, a machine-readable storage medium is provided, wherein machine-executable instructions are stored on the machine-readable storage medium, and when the machine-executable instructions are executed by a processor, the processor is enabled to execute a method or process according to an embodiment of the present disclosure.

[0009] In a fifth aspect of the present disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer-readable storage medium and includes a computer program, which, when executed by a processor of a computer, causes the processor to perform a method or process according to an embodiment of the present disclosure.

[0010] Please note that the summary is provided to introduce a series of concepts in a simplified form, which will be further described in the detailed description below. The summary is not intended to identify key features or essential features of the disclosure, nor is it intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other purposes, features and advantages of the present disclosure will become more clearly understood by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings, in which:

[0012] Figure 1 is a diagram schematically illustrating an example environment in which methods and / or processes according to embodiments of the present disclosure may be implemented;

[0013] Figure 2is a flowchart schematically illustrating a method for generating question-answer data according to an embodiment of the present disclosure;

[0014] Figure 3 A diagram schematically illustrates an example 300 of a workflow for generating question-answer data according to an embodiment of the present disclosure;

[0015] Figure 4 is a diagram schematically illustrating an example process 400 of forward data enhancement according to an embodiment of the present disclosure;

[0016] Figure 5 is a diagram schematically illustrating an example process 500 of reverse data enhancement according to an embodiment of the present disclosure;

[0017] Figure 6 is a diagram schematically illustrating an apparatus for generating question-answer data according to an embodiment of the present disclosure;

[0018] Figure 7 is a schematic block diagram of an example device that may be used to implement embodiments according to the present disclosure.

[0019] Throughout the drawings, same or similar reference numbers generally refer to same or similar elements. DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0021] In the description of the embodiments of the present disclosure, the term "including" and its variations should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects, unless explicitly indicated to be different.

[0022] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0023] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0024] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0025] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0026] As mentioned above, it is necessary to evaluate the capabilities of language models. In the evaluation of language models' capabilities, such as natural language understanding and text-to-structured query language conversion (Text2SQL), accuracy and reliability are extremely important. To achieve this goal, it is necessary to evaluate model performance using high-quality annotated datasets. These annotated datasets must not have problems such as unclear, subjective, ambiguous, unanswerable, or incorrectly annotated, because such degradation problems will distort the evaluation results. For example, if the model gives a "wrong" answer to an extremely ambiguous question, then the question itself should be questioned, not the model's reasoning ability.

[0027] There are still many challenges in the data labeling process. First of all, the labeled answers are very important because they directly affect the evaluation results of the model and the subsequent training direction. The existing labeling methods lack unified standards and specifications, resulting in uneven labeling quality and a very chaotic overall process. This confusion not only affects the quality of the labeled dataset, but also makes the evaluation results unable to accurately reflect the true capabilities of the model. It is expected that the labeled answers will comprehensively cover all possible correct answers, otherwise false negative results are prone to occur, that is, the model is correct but is judged wrong. In addition, it is expected that the answer is "minimal column", which means that the answer should be as concise as possible and contain only necessary columns or attributes to ensure compatibility with the predicted Structured Query Language (SQL) query results, thereby improving the hit rate.

[0028] At present, the vast majority of labeling jobs and labeling result evaluations rely heavily on manual labor. When labeling manually, the labeled answers are often wrong based on factors such as the professional experience of the labelers. In addition, when manually evaluating labeled data, it is often found that some questions have multiple correct answers. In the case of multiple true value answers, it is difficult for humans to pre-label such multiple answers in advance to achieve as much coverage as possible. Therefore, manual labeling is difficult to guarantee accuracy and answer diversity. Manual labeling and evaluation of labeled data is not only time-consuming and labor-intensive, but also very easy to introduce subjectivity and inconsistency in the process, which brings significant instability and affects the accuracy of model evaluation.

[0029] In order to solve at least some of the above and other potential problems, an embodiment of the present disclosure proposes a scheme for generating question and answer data. The scheme includes selecting a first question from questions extracted from a log based on the clarity of the question. The scheme also includes selecting a second question from the first question based on the question meta-information of the first question, and the number of the second questions is less than or equal to the number of the first questions. The scheme also includes assigning a true answer to the second question based on a true answer item from a database. The scheme also includes generating a model answer for the second question based on a model answer item generated by a language model. The scheme also includes associating the second question with one or more answers whose answer differences between the true answer and the model answer meet a difference threshold and storing them to generate a question and answer set based on a comparison between the true answer and the model answer.

[0030] According to the solution for generating question-answer data in the embodiment of the present disclosure, a high-quality annotated data set can be automatically and efficiently generated, ensuring that the questions in the generated question-answer pairs are clearly stated and targeted, while ensuring that the answers in the question-answer pairs fully cover the key points and are both accurate and reliable. The entire process does not require manual intervention or reduces dependence on manual intervention, thereby effectively avoiding the time-consuming and labor-intensive manual participation and possible omissions and errors.

[0031] Reference below Figures 1 to 7 It should be understood that these exemplary embodiments are provided only to enable those skilled in the art to better understand and implement the embodiments of the present disclosure, and are not intended to limit the scope of the present disclosure in any way.

[0032] Figure 1 1 is a schematic diagram schematically illustrating an example environment 100 in which methods and / or processes according to embodiments of the present disclosure may be implemented. Figure 1As shown in , the example environment 100 includes a database 105, a log 110, a computing system 120 including computing nodes (e.g., a locally arranged computing node 121 and a computing node 122 arranged in the cloud), and annotated data 130 (e.g., question-answer pairs). Figure 1 The implementation environment of the embodiment of the present disclosure is described by taking a hybrid computing system as an example. It should be understood that this is only an exemplary and non-restrictive description, and other different types of computing systems are also feasible, such as distributed or centralized computing systems. The appropriate computing environment configuration can be selected according to actual use needs.

[0033] exist Figure 1 In the figure, only limited components and exemplary connection relationships are shown. It should be understood that this is for the purpose of ease of explanation and ease of illustration and is not intended to limit the scope of the present disclosure, and other different components may also exist. For example, a display component and an input component, etc. In an exemplary and non-restrictive manner, the evaluation results for the language model, the evaluation results for the labeled data, etc. can be displayed on the display component, and the questions and answers, etc. can be adjusted through the input component.

[0034] According to an embodiment of the present disclosure, questions can be extracted from log 110, which generally indicates historical data, which may include questions that have been used or asked by users in history, or may also indicate potential problems, etc. Log 110 can be stored in database 105. Database 105 can store historical data such as log 110 and parameter data for language models, etc., and can also serve as a knowledge base for storing knowledge data. The organization and arrangement of knowledge in the database can be based on the type or structure of knowledge data, the desired business direction, etc. In some embodiments, in addition to log 110, questions can be extracted from other different multiple data sources, such as development documents, user feedback records, on-call conversations, etc. In addition, the extraction of questions can be based on predetermined business lines associated with data question-answering applications.

[0035] Questions from log 110 or other sources may be input to a system having computing capabilities (e.g., Figure 1 ), the computing system 120 shown in ), the computing system 120 can be configured to determine the answer corresponding to the question based on the knowledge data from the database 105, and evaluate the determined answer for verification. The verified question-answer pairs (for example, one question corresponds to one answer, one question corresponds to two or more answers, etc.) can be output as the annotated data 130. In the following, the generation and evaluation of the annotated data according to the embodiments of the present disclosure will be further described in detail.

[0036] As described above, at least one computing node included in the computing system 120 can perform processing and operations corresponding to the generation and evaluation of annotation data according to an embodiment of the present disclosure. In an exemplary but non-restrictive manner, computing node 121 can be arranged locally at the user, and computing node 122 can be arranged in the cloud. The cloud can indicate a service model built based on cloud or distributed technology. In this mode, computing resources, storage resources, etc. are coupled together through a network to form a schedulable and scalable resource collection. At least a portion of these resources can be dynamically accessed or allocated to complete tasks without the need to locally own or manage these physical resources.

[0037] A computing node refers to a computing resource or computing power, and may be a device with computing power. For example, a computing node may be provided with a processor and a memory, etc., or may be equipped with a dedicated accelerator (such as a graphics processing unit (GPU)). In addition, a computing node may store and maintain data. In addition, the model (e.g., a large language model) involved in the scheme for generating question and answer data according to an embodiment of the present disclosure may be a model stored on at least one computing node, and of course an external model may also be called.

[0038] like Figure 1 As shown in the example, the local computing node 121 and the computing node 122 in the cloud can be interconnected via a network for communication. Figure 1 Taking the architecture in the example, computing nodes 121 and 122 can communicate via a network to achieve, for example, data synchronization or sharing between nodes. Multiple computing nodes in the example environment 100 can process computing tasks in parallel. In addition, multiple computing nodes in the example environment 100 can have redundancy and fault tolerance mechanisms to ensure reliable execution of computing tasks. When a computing node fails, the system can automatically migrate the job to other nodes that are working normally to ensure the continuity and availability of the task.

[0039] Examples of computing nodes may include supercomputers, personal computers, laptop computers, vehicle-mounted computing devices, mobile devices (such as smart phones, tablet computers, etc.), wearable electronic devices, multimedia devices, personal digital assistants (PDAs), smart home devices, consumer electronic products, or a combination of any one or more of the above devices, etc. It should be understood that the computing nodes described herein are merely exemplary and non-restrictive, and for example, other different types of computing nodes may also be used.

[0040] Combined with the above Figure 1 Describes an example environment in which the methods and / or processes according to embodiments of the present disclosure may be implemented. Figure 2The method 200 for generating question and answer data according to an embodiment of the present disclosure is described. Through the method 200, a high-quality annotated data set can be automatically generated to ensure that the questions are clear and targeted and the answers are comprehensive and accurate.

[0041] Figure 2 is a flowchart schematically illustrating a method 200 for generating question-and-answer data according to an embodiment of the present disclosure. At 210, among the questions extracted from the log, a first question is selected based on the clarity of the question. It is necessary to filter and screen the questions, which not only facilitates the automation of annotation and evaluation by ensuring the quality of the questions, but also helps to improve the processing speed. In the process of selecting the first question, it can be determined whether the statement of the question is clear and specific, and unclear and / or ambiguous questions can be excluded. In addition, an answer convergence threshold dynamically associated with the expected number of questions can also be set, and questions whose possible answers meet the answer convergence threshold are selected.

[0042] At 220, among the first questions, second questions are selected based on the question meta-information of the first questions, and the number of the second questions is less than or equal to the number of the first questions. According to an embodiment of the present disclosure, the first questions obtained at 210 can be further filtered and screened to ensure the convergence of their answers, so that the answers to the questions can be automatically marked and the question and answer pairs can be automatically evaluated. In some embodiments, the question meta-information may include at least one of the following: a question description of the first question, a table field corresponding to the first question, and business knowledge associated with the first question. In this way, the selected second question can clarify the organizational structure of the expected answer and promote the business direction of the question and answer application.

[0043] At 230, a true answer is assigned to the second question based on the true answer item from the database. The true value, also known as the Ground Truth (GT), is a relative standard that is regarded as the true value or close to such a true value. According to an example of the present disclosure, the sample label that has been manually or tool-annotated can be corresponded to the true value. Here, the true answer items stored in the database (also referred to as the knowledge base) can be audited and verified knowledge data with accuracy and reliability. Therefore, the answers assigned (for example, via an annotation tool) based on such true answer items can also inherit a considerable degree of accuracy and reliability. The corresponding true answers that uniquely correspond to each question in the second question are stored in a database, such as a database table.

[0044] At 240, a model answer is generated for the second question based on the model answer items generated by the language model. When processing questions based on natural language descriptions, the language model can capture key information in the text, understand the core points of the question, and thus give targeted answers. For the same question, the language model can run one or more rounds to obtain one or more corresponding model answers. One or more corresponding model answers corresponding to each question in the second question are used to verify the true value answer to the question and expand the coverage of the answer to the question.

[0045] At 250, based on the comparison between the true answer and the model answer, the second question is associated with one or more answers in the true answer and the model answer whose answer difference meets the difference threshold to generate a question and answer set. As described above, for the same question in the second question, its corresponding true answer and model answer are generated differently. The true answer of the corresponding question is compared with the model answer. If the two are consistent (equal or within a predetermined difference threshold), the true answer can be verified as passing the evaluation, and the model answer can be determined as an alternative answer to the question. If the two do not match (not equal or exceed the predetermined difference threshold), it is possible to return to check the correctness of the true answer, and adjust the model answer or discard the model answer. In this way, the question and the corresponding true answer and model answer of the question are stored in association, thereby forming a question and answer set with sample quality.

[0046] According to the method 200 for generating question-answer data in the embodiment of the present disclosure, a high-quality annotated data set can be automatically and efficiently generated, ensuring that the questions in the generated question-answer pairs are clearly stated and targeted, while ensuring that the answers in the question-answer pairs fully cover the key points and are both accurate and reliable. The entire process does not require manual intervention or reduces dependence on manual intervention, thereby effectively avoiding the time-consuming and labor-intensive manual participation and possible omissions and errors.

[0047] Figure 3 A diagram schematically illustrates an example process 300 for generating and evaluating annotated data according to an embodiment of the present disclosure. The example process 300 may include a data collection subprocess 310, an automated question-answering governance subprocess 320, an annotation quality verification subprocess 330, and an optimization and expansion question set subprocess 340. The example process 300 and its subprocesses may be abstracted into an annotation data generation and evaluation unit and corresponding subunits for each subprocess (e.g., a data collection subunit, an automated question-answering governance subunit, etc.), which may be software-based components or systems for generating and evaluating annotated data, and may be implemented on a device with computing capabilities (such as, Figure 1 The system is run on a computing system 120 in the system.

[0048] At the data collection subprocess 310, question data is collected from data sources including logs. The question data will be screened multiple times to filter out questions that are unclear or cannot converge, and retain questions that clarify the organizational structure of the expected answers and promote the business direction of the Q&A application. At the automated Q&A governance subprocess 320, each question will be assigned a true answer and a model answer will be generated as described above, and the true answer will be evaluated and the alternative answer will be supplemented based on the difference between the true answer and the model answer. In the following, this will be further explained in detail in conjunction with the application scenario.

[0049] In the annotation quality verification subprocess 330, the quality of the annotated samples in the question and answer set can be evaluated. A first language model can be used to determine a first accuracy rate for the question and answer pairs in the question and answer set, and a second language model different from the first language model can be used to determine a second accuracy rate for the question and answer pairs in the question and answer set. If the accuracy difference between the first accuracy rate and the second accuracy rate is within the accuracy difference threshold, it can be determined that the annotated samples in the question and answer set meet the quality requirements. If the accuracy difference between the two exceeds the accuracy difference threshold, it can be determined that the annotated samples in the question and answer set have deteriorated beyond the acceptable range, and need to return to 320 for re-management. Here, multi-model accuracy comparison is an example implementation, and can also be achieved by, for example, comparing the manual accuracy rate with the model accuracy rate.

[0050] At the optimization and expansion of the question set subprocess 340, in the question and answer set, the question and answer pairs whose corresponding database tables are unavailable can be deleted at a predetermined frequency, and new question and answer pairs can be added. The question and answer pairs entering the question and answer set are not static, and maintenance and updates will be performed regularly. For example, if the database table corresponding to the question and answer pair is invalid or unavailable, the question and answer pair can be deleted from the question and answer set. In addition, new question and answer pairs can be added from the automated question and answer governance subprocess 320 so that the question and answer set maintains a certain number of labeled samples. It should be understood that Figure 3 The various sub-processes shown in the example are exemplary and not intended to be limiting, and the example process 300 may include different sub-processes for implementing other functions. Figure 4 The generation and evaluation of annotation data according to the embodiments of the present disclosure are further described in detail.

[0051] Figure 4 An example 400 of a workflow for generating and evaluating annotation data according to an embodiment of the present disclosure is schematically illustrated. The example 400 may include topic screening 410, topic annotation 420, annotation evaluation 430, alignment adjustment 440, quality verification 450, and publication delivery 460. In topic screening 410, multiple rounds of screening (such as, Figure 4), for example, the expected question can be selected based on question clarity, answer convergence, question metadata, etc. In question annotation 420, a true answer is assigned to the selected question based on the true answer items in the database, and these true answer items can be knowledge data that have been reviewed and verified.

[0052] At the annotation evaluation 430, the annotation data is evaluated based on the difference between the true answer and the model answer. The language model generates one or more model answers for the question. By using the output of the language model to evaluate the annotation data, on the one hand, the correctness of the assigned true answer can be verified, and on the other hand, the generated model answer can be used to enrich the coverage of the answer. At the answer adjustment 440, in response to the answer difference between the true answer and the model answer not meeting the difference threshold, the correctness of the true answer can be checked, for example, the understanding of the question is aligned and the true answer is adjusted, and the model answer can also be adjusted or discarded. After being adjusted, the true answer and the model answer need to be returned to the annotation evaluation 430 for re-evaluation until they pass.

[0053] At quality check 450, the question-answer pairs in the question set may be checked. Different language models may be used to determine the first correct rate and the second correct rate for the question-answer pairs in the question-answer set, or artificial and language models may be used to determine the first correct rate and the second correct rate for the question-answer pairs in the question-answer set. The quality of the labeled data in the question-answer set may be evaluated based on the degree of difference between the correct rates. It should be understood that Figure 4 Two checks (i.e., the first check and the second check) are shown in FIG. More or fewer checks are also feasible, and the number of checks can be determined according to actual needs. If the question and answer set passes the quality check, then at release delivery 460, the question and answer set is distributed to the user for a predetermined purpose, such as verifying the model capability, the performance of the question and answer application, etc.

[0054] Figure 5 A diagram schematically illustrates an example process 500 of annotated data verification based on answer differences according to an embodiment of the present disclosure. It should be understood that the application scenario of Text2SQL is used for illustration herein, but the embodiments of the present invention are not limited to this reference scenario and can also be used in other different application scenarios. In an embodiment of the present disclosure, for each question in the second question, the corresponding true answer of the corresponding question (e.g., question 510) includes a visual uniform resource locator (URL) 520, and the corresponding model answer of the corresponding question includes a first type of model SQL statement 530 (e.g., model ANSI-SQL).

[0055] According to an embodiment of the present disclosure, based on the visualization URL 520 included in the corresponding truth value answer, a structured presentation 522 (also referred to as a truth value structured presentation) of the corresponding question (e.g., question 510), a second type of truth value SQL statement 524 (e.g., truth value CH-SQL), and a truth value execution result 526 can be determined, wherein the structured presentation 522 can be at least one data attribute presented in, for example, a capsule component. For the same question (e.g., question 510), the visualization URL 520 in the truth value answer can be stored in a database table in association with the second type of truth value SQL statement 524 and the truth value execution result 526, and the second type of truth value SQL statement 524 and the truth value execution result 526 corresponding to the visualization URL 520 can be provided simultaneously through, for example, an annotation tool. Annotating the visualization URL 520 is more efficient, and the execution result can be visually viewed through the capsule drag operation, which is more convenient and quicker to view the execution result than annotating SQL.

[0056] According to an embodiment of the present disclosure, based on the first type of model SQL statement 530, a second type of model SQL statement 532 (e.g., model CH-SQL) and a model execution result 534 of a corresponding question (e.g., question 510) can be determined. For the same question (e.g., question 510), each time the language model is run, a first type of model SQL statement can be provided, and the first type of model SQL statement has a corresponding second type of model SQL statement and model execution result. In some embodiments, for the same question, the language model can be run multiple times to obtain multiple model answers, and the multiple model answers may be different.

[0057] According to an embodiment of the present disclosure, by comparing the second type of true value SQL statement 524 with the second type of model SQL statement 532, the first difference between the corresponding true value answer and the corresponding model answer can be determined. In addition, by comparing the true value execution result 526 with the model execution result 534, the second difference between the corresponding true value answer and the corresponding model answer is determined. The answer difference between the corresponding true value answer and the corresponding model answer can be determined based on the first difference and the second difference. In some embodiments, the difference between the corresponding true value answer and the corresponding model answer can be determined based on at least one of the first difference and the second difference, or the difference between the corresponding true value answer and the corresponding model answer can be determined based on the weighted average of the first difference and the second difference, and so on.

[0058] According to an embodiment of the present disclosure, in response to the first difference satisfying the first difference threshold and the second difference satisfying the second difference threshold, the corresponding true value answer can be determined as the corresponding accurate answer that passes the evaluation, and the corresponding model answer can be determined as the corresponding alternative answer to the corresponding question. In addition, in response to at least one of the first difference and the second difference not satisfying the corresponding difference threshold in the first difference threshold and the second difference threshold, the correctness of the corresponding true value answer can be checked, and the corresponding model answer can be adjusted, or the corresponding model answer can be deleted. In some embodiments, if the first type of model SQL statement 530 fails to run, or if the true value execution result 526 is inconsistent with the model execution result 534, manual intervention can be performed to check the correctness of the first type of model SQL statement 530.

[0059] Figure 6 Schematically illustrates an apparatus 600 for generating question-answer data according to an embodiment of the present disclosure. The apparatus 600 may include multiple units or modules for performing the steps or actions in the method or process discussed above. Figure 6 As shown in , the device 600 includes: a first selection module 610, configured to select a first question among the questions extracted from the log based on the question clarity of the question; a second selection module 620, configured to select a second question among the first questions based on the question meta-information of the first question, and the number of the second questions is less than or equal to the number of the first questions; an answer assignment module 630, configured to assign a true answer to the second question based on a true answer item from a database; an answer generation module 640, configured to generate a model answer for the second question based on a model answer item generated by a language model; and a question set generation module 650, configured to associate the second question with one or more answers whose answer difference between the true answer and the model answer meets a difference threshold and store them to generate a question and answer set based on a comparison between the true answer and the model answer.

[0060] In some embodiments, wherein for each of the second questions, the corresponding truth answer to the corresponding question includes a visualization uniform resource locator URL, and the corresponding model answer to the corresponding question includes a first type of model SQL statement, the device 600 further includes: a first determination module, configured to determine the truth structured presentation, the second type of truth SQL statement and the truth execution result of the corresponding question based on the visualization URL; and a second determination module, configured to determine the second type of model SQL statement and the model execution result of the corresponding question based on the first type of model SQL statement.

[0061] In some embodiments, the corresponding true answer to the corresponding question includes a true answer stored in the database, and the model answer to the corresponding question includes multiple model answers obtained by running the language model for multiple rounds.

[0062] In some embodiments, for each of the second questions, the apparatus 600 further comprises:

[0063] a first comparison module configured to determine a first difference between the corresponding true value answer and the corresponding model answer by comparing the second type of true value SQL statement with the second type of model SQL statement; a second comparison module configured to determine a second difference between the corresponding true value answer and the corresponding model answer by comparing the true value execution result with the model execution result;

[0064] A difference determination module is configured to determine the answer difference between the corresponding true answer and the corresponding model answer based on the first difference and the second difference.

[0065] In some embodiments, the device 600 also includes an evaluation module, which is configured to determine the corresponding true answer as the corresponding accurate answer that passes the evaluation in response to the first difference satisfying a first difference threshold and the second difference satisfying a second difference threshold, and determine the corresponding model answer as the corresponding alternative answer to the corresponding question.

[0066] In some embodiments, the evaluation module is also configured to check the correctness of the corresponding true value answer, adjust the corresponding model answer, or delete the corresponding model answer in response to at least one of the first difference and the second difference not satisfying the corresponding difference threshold between the first difference threshold and the second difference threshold.

[0067] In some embodiments, the apparatus 600 further comprises an optimization module configured to delete, from the question and answer set, question and answer pairs whose corresponding database table is unavailable at a predetermined frequency, and to supplement with new question and answer pairs.

[0068] In some embodiments, the questions are extracted from one or more other different data sources in addition to the logs, and the extraction of the questions is based on a predetermined business line associated with the data questioning application.

[0069] In some embodiments, the question meta-information includes at least one of the following: a question description of the first question; a table field corresponding to the first question; and business knowledge associated with the first question.

[0070] Figure 7 FIG. 1 is a block diagram of an electronic device 700 according to some embodiments of the present disclosure. The device 700 may be a device or apparatus described in an embodiment of the present disclosure. Figure 7 As shown, the device 700 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 701, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 702 or loaded from a storage unit 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The CPU / GPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704. Although not shown in FIG. Figure 7 As shown in FIG. 7 , device 700 may further include a coprocessor.

[0071] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0072] The various methods or processes described above may be performed by the CPU / GPU 701. For example, in some embodiments, the methods may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the CPU / GPU 701, one or more steps or actions in the methods or processes described above may be performed.

[0073] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0074] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0075] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0076] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, and conventional procedural programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby implementing various aspects of the present disclosure.

[0077] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0078] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0079] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the equipment, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous frames can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the functions involved. It should also be noted that each frame in the block diagram and / or flow chart, and the combination of frames in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0080] Various embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the various embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various embodiments disclosed herein.

Claims

1. A method for generating question-answering data, comprising: Among the questions extracted from the log, selecting a first question based on question clarity of the questions; Among the first questions, second questions are selected based on question meta information of the first questions, and the number of the second questions is less than or equal to the number of the first questions; Assigning truth answers to the second questions based on truth answer items from a database, wherein the corresponding truth answer to each of the second questions is a visualization uniform resource locator URL, wherein a second type of truth SQL statement and a truth execution result of the corresponding question are determined based on the visualization URL; Generate a model answer for the second question based on a model answer item generated by the language model, the model answer comprising a plurality of different answers generated by running the language model for a plurality of rounds for the same question in the second questions, the corresponding model answer for each question in the second questions being a first type of model SQL statement, wherein the second type of model SQL statement and the model execution result of the corresponding question are determined based on the first type of model SQL statement; as well as Based on the comparison between the true answer and the model answer, the second question is associated with one or more answers in the true answer and the model answer whose answer difference satisfies a difference threshold and is stored to generate a question-answer set, wherein determining the answer difference comprises: Determine a first difference between the corresponding true answer and the corresponding model answer by comparing the second type of true SQL statement corresponding to the visualization URL with the second type of model SQL statement corresponding to the first type of model SQL statement; Determining a second difference between the corresponding true value answer and the corresponding model answer by comparing the true value execution result corresponding to the visualization URL with the model execution result corresponding to the first type of model SQL statement; and The answer difference between the corresponding true answer and the corresponding model answer is determined based on the first difference and the second difference.

2. The method according to claim 1, further comprising: Based on the visualization URL, a truth structured representation of the corresponding question is determined.

3. The method according to claim 1, wherein: The corresponding true answer to the corresponding question includes a true answer stored in the database.

4. The method according to claim 1, further comprising: In response to the first difference satisfying a first difference threshold and the second difference satisfying a second difference threshold, Determine the corresponding true value answer as the corresponding accurate answer that passes the evaluation, and The corresponding model answer is determined as the corresponding candidate answer to the corresponding question.

5. The method according to claim 1, further comprising: In response to at least one of the first difference and the second difference not satisfying a corresponding difference threshold of a first difference threshold and a second difference threshold, Check the correctness of the corresponding truth answer, and Adjust the corresponding model answer, or delete the corresponding model answer.

6. The method according to claim 1, further comprising: In the question-answer set, question-answer pairs whose corresponding database table is unavailable are deleted at a predetermined frequency, and new question-answer pairs are added. 7 . The method according to claim 1 , wherein the questions are extracted from one or more other different data sources in addition to being extracted from the log, and the extraction of the questions is based on a predetermined business line associated with a data question answering application.

8. The method according to claim 1, wherein the question meta-information comprises at least one of the following: A description of the problem of the first problem; Table fields corresponding to the first question; and Business knowledge associated with the first question.

9. A device for generating question-answer data, comprising: A first selection module configured to select a first question among the questions extracted from the log based on the question clarity of the questions; A second selection module is configured to select second questions from among the first questions based on question meta-information of the first questions, the number of the second questions being less than or equal to the number of the first questions; an answer assignment module configured to assign truth answers to the second questions based on truth answer items from a database, the corresponding truth answer for each of the second questions comprising a visualization uniform resource locator URL, wherein a truth SQL statement of a second type and a truth execution result of the corresponding question are determined based on the visualization URL; an answer generation module configured to generate a model answer for the second question based on a model answer item generated by a language model, the model answer comprising a plurality of different answers generated by running the language model for a plurality of rounds for the same question in the second questions, the corresponding model answer for each question in the second questions comprising a first type of model SQL statement, wherein the second type of model SQL statement and the model execution result of the corresponding question are determined based on the first type of model SQL statement; as well as A question set generation module is configured to associate the second question with one or more answers whose answer differences between the true answer and the model answer meet a difference threshold and store them to generate a question and answer set based on a comparison between the true answer and the model answer, wherein determining the answer difference comprises: Determine a first difference between the corresponding true value answer and the corresponding model answer by comparing the second type of true value SQL statement corresponding to the visualization URL with the second type of model SQL statement; Determining a second difference between the corresponding true value answer and the corresponding model answer by comparing the true value execution result with the model execution result; and The answer difference between the corresponding true answer and the corresponding model answer is determined based on the first difference and the second difference.

10. An electronic device comprising: processor; as well as A memory coupled to the processor, wherein instructions are stored in the memory, and when the instructions are executed by the processor, the processor is caused to perform the method according to any one of claims 1 to 8.

11. A machine-readable storage medium having machine-executable instructions stored thereon, wherein the machine-executable instructions, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 8.

12. A computer program product comprising a computer program which, when executed by a processor of a computer, causes the processor to perform the method according to any one of claims 1 to 8.

Images