Evaluation data generation method and device, storage medium and program product

Abstract

The embodiment of the invention provides an evaluation data generation method and device, a storage medium and a program product. In the embodiment of the invention, the cue word for guiding the data generation model to generate the evaluation data is generated through the metadata of the data table in the data source, and the evaluation data set for evaluating the question and answer application of the data source is generated by utilizing the data generation model according to the cue word, so that automatic generation of the evaluation data set is realized; compared with an evaluation data set output mode of manual annotation, the generation efficiency of the evaluation data set can be improved. On the other hand, the metadata of the data table can reflect the structure and the data characteristics of the data table, so that the cue word is generated according to the metadata of the data table, the cue word can be closely associated with the data table, and then the evaluation data generated according to the cue word can be closely associated with the data table of the data source; and the validity of the generated evaluation data set can be ensured.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an evaluation data generation method, device, storage medium and program product. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the advancement of Large Language Models (LLMs), intelligent applications are widely used in various industries. Among them, Data Question Answering (DQA) applications are based on LLMs and are designed to enable non-technical personnel to interact directly with complex data sets through natural language, so as to easily obtain the required information. This capability greatly reduces the threshold for data analysis and improves decision-making efficiency.

[0003] In order to improve the efficiency of user database query, data question and answer applications are applied to database queries. Users only need to provide query requirements described in natural language, and the data question and answer application can automatically convert the query requirements described in natural language into programming language query statements supported by the database, such as Structured Query Language (SQL) query statements. Furthermore, the question and answer model can use the query statements to perform corresponding query operations in the database and obtain query results. This query method does not require users to master programming languages, which can lower the threshold for data query.

[0004] Evaluating data question-and-answer applications is a key step in improving and optimizing the performance of data question-and-answer applications, and an objective, complete, reliable, and closely integrated evaluation dataset is the core of a good evaluation system. Evaluation datasets are mainly derived from manual annotations, or are expanded using datasets with manual intervention as seeds, which makes the output efficiency of evaluation datasets low. Summary of the invention

[0005] Multiple aspects of the present application provide a method, device, storage medium and program product for generating evaluation data, so as to realize automatic generation of evaluation data and improve the output efficiency of evaluation data.

[0006] The present application embodiment provides a method for generating evaluation data, including: Get the metadata of the data table in the data source; Generate prompt words according to the metadata; the prompt words are used to guide the pre-trained data generation model to generate evaluation data; The data generation model is used to generate an evaluation data set for evaluating the question-and-answer application of the data source according to the prompt word.

[0007] The embodiment of the present application further provides an electronic device, comprising: a memory and a processor; wherein the memory is used to store a computer program; The processor is coupled to the memory and configured to execute the computer program to perform the steps in the aforementioned data generating method.

[0008] The present application also provides a method for generating evaluation data, including: In response to a request to call a target service, determining a processing resource corresponding to the target service; the target service is a service that provides a method for generating evaluation data; The processing resources are used to execute the steps in the aforementioned evaluation data generation method.

[0009] The embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the aforementioned evaluation data generating method.

[0010] An embodiment of the present application further provides a computer program product, including a computer program. When the computer program is executed by one or more processors, the one or more processors are caused to execute the steps in the aforementioned evaluation data generating method.

[0011] In the embodiment of the present application, the metadata of the data table in the data source is used to generate prompt words for guiding the data generation model to generate evaluation data, and the data generation model is used to generate an evaluation data set for evaluating the question and answer application of the data source according to the prompt words, thereby realizing the automatic generation of the evaluation data set. Compared with the manual annotation evaluation data set output method, the generation efficiency of the evaluation data set can be improved. On the other hand, since the metadata of the data table can reflect the structure and data characteristics of the data table, the prompt words generated according to the metadata of the data table can make the prompt words closely associated with the data table, and then the evaluation data generated according to the prompt words can be closely associated with the data table of the data source, which can ensure the validity of the generated evaluation data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A schematic diagram of a flow chart of a data generation method provided in an embodiment of the present application; Figure 2 A schematic diagram of a specific process of the data generation method provided in an embodiment of the present application; Figure 3 A schematic diagram of a flow chart of another data generation method provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0014] It should be noted that, in the case where the embodiments of the present application involve user information, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0015] The concepts or terms involved in the embodiments of the present application are explained below.

[0016] Large Language Model (LLM): LLM is an AI model designed to understand and generate human language. They are trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more.

[0017] Data questioning application or system: A data questioning application or system is a solution that combines natural language processing (NLP) and database query technology. It allows users to ask questions in natural language and automatically converts these questions into query languages ​​supported by the database (usually SQL query statements). After that, the data questioning application or system executes the corresponding query statements in the database to obtain the answers to the questions. The core goal of this technology is to lower the threshold for data access through a natural language interface, so that non-technical personnel can also efficiently extract valuable information from data.

[0018] Evaluation system: An evaluation system is a set of tools and methods used in computer science and technology to evaluate the performance of software systems, algorithms, models or applications. The goal of the evaluation system is to measure all aspects of the system through a series of standard tests and evaluation methods to ensure that it meets the expected functional, performance and quality requirements.

[0019] Database: A database is an ordered collection of structured information or data, generally stored in electronic form in a computer system.

[0020] Metadata: Metadata is data that describes data, which is information about the organization of data, data domains and their relationships.

[0021] Evaluation dataset: refers to a set of data used to evaluate and verify the technical performance of machine learning models, natural language processing systems, computer vision models, etc. Evaluation datasets are usually used to test the performance of artificial intelligence (AI) models on unseen data to evaluate their generalization ability and actual application effects.

[0022] Prompt word: refers to the input text used to guide the language model to generate a specific type of output.

[0023] Prompt word engineering: refers to the process of designing and optimizing prompt words, with the goal of enabling the language model to generate high-quality output that meets expectations.

[0024] In order to improve the generation efficiency of evaluation data, in some embodiments of the present application, prompt words for guiding the data generation model to generate evaluation data are generated through the metadata of the data table in the data source, and the data generation model is used to generate an evaluation data set for evaluating the question and answer application of the data source according to the prompt words, thereby realizing the automatic generation of the evaluation data set. Compared with the manual annotation evaluation data set output method, the generation efficiency of the evaluation data set can be improved. On the other hand, since the metadata of the data table can reflect the structure and data characteristics of the data table, the prompt words generated according to the metadata of the data table can make the prompt words closely associated with the data table, and then the evaluation data generated according to the prompt words can be closely associated with the data table of the data source, which can ensure the validity of the generated evaluation data set.

[0025] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0026] It should be noted that the same reference numerals represent the same object or the same step in the following drawings and embodiments, and therefore, once an object or a step is defined in one drawing or embodiment, there is no need to further discuss it in the subsequent drawings and embodiments.

[0027] Figure 1 The following is a flow chart of the method for generating evaluation data provided in the embodiment of the present application. Figure 1 As shown, the evaluation data generation method mainly includes the following steps: 101. Get the metadata of the data table in the data source.

[0028] 102. Generate prompt words based on the metadata; the prompt words are used to guide the pre-trained data generation model to generate an evaluation data set.

[0029] 103. Generate an evaluation data set for evaluating the question-and-answer application of the data source according to the prompt words using the data generation model.

[0030] In this embodiment, the data source refers to a data storage system that supports storing data in the form of data tables. The data source may be a database, a data warehouse, or a pre-stored data table set.

[0031] In order to lower the access threshold of the data source, a question-and-answer application can be designed for the data source. The question-and-answer application supports users to describe query requirements (i.e., questions) in natural language, and converts the query requirements described in natural language into query statements written in a programming language supported by the data source. The query statement can be a query statement written in a structured query language (SQL), a link cache query language (LQL), or a domain-specific language (DSL). The general query statement is an SQL query statement. Furthermore, the question-and-answer application can use the query statement to query the data source to obtain a query result that meets the user's query requirements, that is, the answer to the question (i.e., query requirement) submitted by the user. In the embodiment of the present application, the specific implementation form of the question-and-answer application is not limited. The question-and-answer application can be implemented as a client, an application (APP), a World Wide Web (Web) web page or a mini-program, etc.

[0032] Using evaluation data to evaluate the performance of the question-answer application is a key link in optimizing the performance of the question-answer application. In the present embodiment, in order to automatically generate evaluation data, considering that the metadata of the data table can reflect the characteristics of the data in the data table to a certain extent, the evaluation data for evaluating the question-answer application of the data source is constructed according to the metadata of the data table, which can be closely associated with the data in the data table. The accuracy of the evaluation can be improved by evaluating the question-answer application of the data source using evaluation data that is closely associated with the data in the data source. Based on this, in step 101 of the present embodiment, the metadata of the data table in the data source can be obtained. The metadata of the data table refers to the data that describes the structural information of the data table. It includes the definition of the table, the attributes of the field, the data type, the relationship constraints and the index, etc., but is not limited to this.

[0033] In this embodiment, the specific method of obtaining the metadata of the data table is not limited. In some embodiments, the metadata of the data table in the data source can be obtained in advance; and the metadata of the data table is stored in the metadata management system. Accordingly, step 101 can be implemented as: obtaining the metadata of the data table in the data source from the metadata management system corresponding to the data source.

[0034] The metadata of a data table may include basic information of the data table. The basic information of a data table refers to information used to describe the properties of the data table, and may include: the name of the data table, the structure information of the data table, and the index information of the data table. Based on this, the basic information of the data table may be obtained as the metadata of the data table. Since the basic information of a data table reflects the properties of the data table, obtaining the basic information of the data table may provide a basis for the subsequent construction of evaluation data for evaluating the question-and-answer application of the data source.

[0035] The system table of the data source stores some metadata of the data table. The system table is part of the database management system and stores important metadata about the database structure and configuration. These metadata describe the data objects in the database. The data objects in the database may include: data tables, fields, views, columns, indexes and user permissions contained in the data tables, etc., and are crucial for managing and optimizing the database. Based on this, the basic information of the data table can be obtained from the system table of the data source. For example, the database connection tool can be used to obtain the basic information of the data table from the system table of the data source. Among them, the database connection tool refers to a class of software applications that allow users or developers to interact with the database. Such tools can be graphical or command line interface, and usually support multiple database management systems. Through these tools, users can perform SQL queries, manage data objects, and import and export data.

[0036] In this embodiment, the basic information of the data table is directly obtained from the system table of the data source as metadata, which can simplify the process of obtaining the basic information of the data table and improve the efficiency of obtaining the basic information of the data table.

[0037] In other embodiments, the metadata of the data table also includes: field information of the data table. Accordingly, the field information of the data table can be obtained as the metadata of the data table. Since the field information of the data table can reflect the characteristics of the fields in the data table, obtaining the field information of the data table as the metadata of the data table can provide a basis for the subsequent construction of evaluation data for evaluating the question and answer application of the data source, and strengthen the correlation between the constructed evaluation data and the data in the data source.

[0038] In some embodiments, the field information of a data table may include: structural information of the fields of the data table and data characteristics of the fields. The structural information of a field refers to information describing the structural attributes of the field, and may include: field name, field type, primary key and foreign key, etc. The data characteristics of the field value also play an important role in constructing evaluation data. Among them, the data characteristics of the field are used to reflect the characteristics and distribution of the field value.

[0039] Therefore, the field structure information of the data table can be obtained from the system table of the data source; and the data characteristics of the fields of the data table can be obtained from the data source, and the field structure information and field data characteristics can be used as field information. Among them, obtaining the field structure information and field data characteristics as metadata of the data table can provide a basis for the subsequent construction of evaluation data for evaluating the question and answer application of the data source. The field structure information and field data characteristics can reflect the characteristics and distribution of the field value. Therefore, the field structure information and field data characteristics as metadata can further strengthen the correlation between the constructed evaluation data and the data in the data source.

[0040] In some embodiments, the data features of a field may include: a value domain of the field. The value domain of the field is used to limit the value range of the field. Accordingly, the value domain of the field of the data table can be obtained from the data source as the data feature of the field of the data table. Since the value domain of the field limits the value range of the field, the value domain of the field is used as metadata to provide a basis for constructing evaluation data for evaluating the question and answer application of the data source, so that the constructed evaluation data can be closely associated with the data in the data source.

[0041] In some embodiments, since some data sources have a large amount of data, that is, they contain more data tables and each field has more values, a sampling method can be used to obtain the value range of the field. Accordingly, for any field A of the data table, when the amount of data in the data source is greater than or equal to the set data amount threshold, the value of field A contained in the data source can be sampled according to the set sampling ratio to obtain the sample value of field A. For example, the value of field A contained in the data source can be randomly sampled according to the set sampling ratio to obtain the sample value of the set sampling ratio as the sample value of field A. Alternatively, the value of field A contained in the data source can be sampled according to the set sampling method to obtain the sample value of field A. The set sampling method can sample every set number of values, or start sampling from the value at the set position until the sample value reaches the set number threshold, and so on.

[0042] After obtaining the sample value of field A, the value range of field A can be determined according to the sample value of field A. For a field where field A is a numeric type, the maximum sample value and the minimum sample value can be obtained from the sample value of field A, and the numerical range defined by the minimum sample value and the maximum sample value is used as the value range of field A. For a field where field A is a non-numeric type, different sample values ​​can be obtained from the sample value of field A, and the sample value set formed by different sample values ​​is used as the value range of field A. In this embodiment, by sampling a field with a large amount of data to obtain the value range of field A, the amount of data for determining the value range of field A can be reduced, and the efficiency of determining the value range of field A can be improved. On the other hand, by sampling a field with a large amount of data to obtain the value range of field A, the value of the field itself is also taken into account, and the accuracy of the obtained value range of field A can be ensured to a certain extent.

[0043] Accordingly, when the data volume of the data source is less than the set data volume threshold, the value of field A can be obtained from the data source; and the value range of field A can be determined according to the value of field A. For the specific implementation of determining the value range of field A according to the value of field A, please refer to the above-mentioned content related to determining the value range of field A according to the sample value of field A, which will not be repeated here. Since the data volume of the data source is small, the value range of field A can be directly determined according to the value of field A recorded in the data table for a data source with a small data volume, while taking into account the efficiency of determining the value range of field A, the accuracy of the determined value range of field A can be ensured.

[0044] For a field, its data characteristics may also include: the sampling value of the field. Accordingly, for any field A, the value of field A contained in the data table can be sampled according to the set sampling ratio to obtain the sample value of field A as the data characteristic of field A. For the convenience of description and distinction, the sampling ratio used to extract the data characteristics of field A here is defined as the first sampling ratio; the sampling ratio used for sampling when determining the value range of field A is defined as the second sampling ratio. The first sampling ratio and the second sampling ratio may be the same or different. For example, the first sampling ratio may be smaller than the second sampling ratio, etc.

[0045] In some embodiments, the data characteristics of a field may also include: statistical characteristics of the field. The statistical characteristics of any field A may include: one or more of the maximum value of field A, the minimum value of field A, the average value of field A, the mean square error of field A, the variance of field A, and the standard deviation of field A. Multiple refers to more than 2 (including 2). Since the statistical characteristics of a field reflect the distribution characteristics of the value of the field to a certain extent, using the statistical characteristics of the field as metadata to provide a basis for constructing evaluation data for evaluating the query and answer application of the data source can make the constructed evaluation data closely related to the data in the data source.

[0046] In some embodiments, the data features of a field may also include: outliers of the field. Outliers refer to data in the value of a field that is significantly different from other values ​​of the field. Accordingly, for any field A, the outliers of field A can be determined based on the value of field A as the data features of field A. In the embodiments of the present application, the specific implementation method of determining the outliers of field A based on the value of field A is not limited. Since the outliers of a field reflect the special circumstances of the field, the special circumstances of the field are used as metadata to provide a basis for constructing evaluation data for evaluating the query and answer application of the data source, so that the constructed evaluation data can be closely associated with the data in the data source.

[0047] Optionally, the interquartile range (IQR) method can be used to determine outliers from the values ​​of field A. Specifically, the values ​​of field A are sorted to obtain the sorting result of field A; the first quartile Q1 and the third quartile Q3 are determined from the sorting result of field A. The first quartile can also be called the 25th percentile. The third quartile (Q3) can also be called the 75th percentile. Further, the interquartile range IQR can be determined from the first quartile (Q1) and the third quartile (Q3). IQR=Q3-Q1. Afterwards, the target range can be determined based on the interquartile range IQR, the first quartile Q1 and the third quartile Q3. The upper limit of the target range is equal to the third quartile Q3 plus the set multiple of the four-quartile range; the lower limit of the target range is equal to the first quartile Q1 minus the set multiple of the four-quartile range. The multiple is set to be greater than 1, generally 1.5, so the upper limit of the target range is equal to Q3+1.5 * IQR4; the lower limit of the target range is equal to Q1-1.5 * IQR. Further, the outliers of field A can be determined based on the target range. Specifically, the values ​​of field A that do not belong to the target range can be determined as outliers.

[0048] Alternatively, the mean and standard deviation of the values ​​of field A may be calculated. For any value of field A, if the absolute value of the difference between the value and the mean of field A is greater than a set multiple of the standard deviation, the value is determined to be an outlier. In this manner, the set multiple is generally greater than or equal to 3. The outlier determination method shown in the above embodiment is only an exemplary description and does not constitute a limitation.

[0049] In addition to outliers, a field may also have some special values ​​that meet the set conditions, such as outliers with a high frequency of occurrence, or data that follows a certain pattern, such as data that appears periodically, etc. Based on this, the target value that meets the set conditions can also be determined from the value of any field A as the data feature of field A. Among them, the set conditions can be implemented as: the frequency of occurrence is greater than or equal to the set frequency threshold; or, the same value appears periodically; or, the same value appears at a specific time, etc., but are not limited to this. Since the outliers of a field reflect the special circumstances of the field, the special circumstances of the field are used as metadata to provide a basis for constructing evaluation data for evaluating the question-and-answer application of the data source, so that the constructed evaluation data can be closely associated with the data in the data source.

[0050] It is worth noting that the data characteristics of a field may include some or all of the value range, sample value, statistical characteristics, outliers and target value of the aforementioned field, which is mainly determined by the distribution of the field value. Generally, the data characteristics of a field include the value range and sample value of the field. Whether the data characteristics of a field include statistical characteristics, outliers and target values ​​depends on the distribution of the field value. For example, if there are no outliers and target values ​​in the value of a certain field, the data characteristics of the field do not include outliers and target values. For another example, if the value of a certain field is non-numeric data, the data characteristics of the field do not include statistical characteristics, etc.

[0051] The metadata of the data table shown in the above embodiment is only an example and does not constitute a limitation. Figure 2 As shown, the metadata of the data table can be stored in the metadata management system of the data source. In this way, when constructing the evaluation data set, the metadata of each data table in the data source can be obtained from the metadata management system. For example, a query statement (such as an SQL query statement) written in a programming language supported by the metadata management system can be used to obtain the metadata of each data table in the data source from the metadata management system.

[0052] In other embodiments, when constructing an evaluation data set for a question-and-answer application of a data source, metadata of each data table in the data source may be obtained in real time. The specific acquisition method may refer to the relevant content of the aforementioned embodiment and will not be described in detail here.

[0053] Further, if Figure 2As shown, a metadata base table can be generated based on the metadata of the acquired data table. The base table is a physical table that stores metadata. It is the basic structure for directly storing data in the database, in contrast to views, temporary tables, or other derived data representations. The data in the base table can be directly queried, inserted, updated, or deleted, and is the basis for all data operations. Specifically, the metadata of the same metadata type is stored in the same column according to the metadata type, and the metadata base table is obtained.

[0054] In an embodiment of the present application, in order to automatically generate an evaluation data set for evaluating the question-answering application of a data source, a data generation model is introduced. The data generation model is a pre-trained generative model, such as a generative adversarial network (GAN) model, a self-attention mechanism model, or a bidirectionally encoded self-attention mechanism model, but is not limited thereto. In this embodiment, a piece of evaluation data may include: a query requirement described in natural language, i.e., a question. Therefore, the training samples of the data generation model may include: a question sample. The question sample may be a natural language describing the query requirement for querying a data source. For example, "What is the total annual sales volume of a certain product?" Another example is, "Please give the performance indicators of a certain server", etc. The data generation model can learn the description methods of these question samples during the training process.

[0055] After the data generation model is trained, it can generate evaluation data based on the prompt words (Prompt) obtained to guide the data generation model to generate evaluation data. Among them, the prompt word (Prompt) in the field of natural language processing (NLP) and artificial intelligence refers to a piece of text or instruction provided to the model to guide the model to generate a specific type of output or perform a specific task. In this embodiment, the prompt word is used to guide the data generation model to generate evaluation data.

[0056] Since the metadata of the data table reflects the characteristics of the data table itself and the characteristics of the data in the data table, in step 102, prompt words for guiding the data generation model to generate evaluation data can be generated according to the metadata of the data table. In the embodiment of the present application, the specific implementation method of generating prompt words according to the metadata of the data table is not limited.

[0057] In some embodiments, a prompt word template can be pre-designed. For example, the prompt word template may be: "Please generate evaluation data based on the metadata of the following data table. The metadata of the data table is: [table name], [table structure], [field name], [type], [primary key of the field], [foreign key of the field], [value range of the field], [sample value of the field], [statistical characteristics of the field], [outliers of the field] and [target value of the field]". Among them, "[ ]" is represented by a placeholder. For any data table B, based on the information represented by the placeholder in the prompt word template, the metadata of the data table B can be written into the corresponding placeholder to obtain the prompt word corresponding to the data table B. Of course, based on the information represented by the placeholder in the prompt word template, the metadata of multiple data tables can also be written into the corresponding placeholder to obtain the prompt words of multiple data tables. This prompt word is suitable for associated queries.

[0058] In actual application, some data sources have many data tables and large data volumes. In order to improve the efficiency of prompt word generation, Figure 2 As shown in "Metadata Sampling and Segmentation", the metadata of the data table can be sampled to obtain metadata samples. For example, the metadata of the data table can be randomly sampled according to a set sampling ratio; or, the metadata of the data table can be sampled according to a set sampling method. For the specific implementation of sampling the metadata of the data table according to the set sampling method, please refer to the relevant content of sampling the value of field A mentioned above, which will not be repeated here.

[0059] Alternatively, the target sampling ratio can be determined based on the number x0 of evaluation data in the evaluation data set to be generated and the amount of data in the data table. Specifically, the amount y0 of source data required to generate the number x0 of evaluation data required for the evaluation data set can be determined based on the functional relationship between the number x of evaluation data and the amount y of required source data; then, the amount y0 of evaluation data required to generate the number x0 of evaluation data sets required can be calculated as the ratio of the amount of data in the data table to the amount of data in the data table, which is used as the target sampling ratio. The functional relationship between the number x of evaluation data and the amount y of required source data is obtained by the inventor of the present application through preliminary experimental analysis.

[0060] After determining the target sampling ratio, the metadata of the data table can be sampled according to the target sampling ratio to obtain metadata samples. In this embodiment, the target sampling ratio is determined according to the number x0 of evaluation data in the evaluation data set to be generated and the data volume of the data table, so that the data volume of the sampled metadata samples can be adapted to the number x0 of evaluation data required by the evaluation data set to be generated. On the one hand, it can prevent the sampling ratio from being too small, and the sampled metadata samples cannot support the generation of the corresponding number x0 of evaluation data, resulting in the evaluation data set failing to reach the expected scale. On the other hand, it can prevent the sampling ratio from being too large and reducing the efficiency of subsequent prompt word generation.

[0061] After obtaining the metadata sample, a prompt word can be generated according to the metadata sample. Specifically, for the metadata sample corresponding to any data table B, based on the information represented by the placeholder in the prompt word template, the metadata sample of the data table B can be written into the corresponding placeholder to obtain the prompt word corresponding to the sample of the data table B. Of course, based on the information represented by the placeholder in the prompt word template, metadata samples of multiple data tables can also be written into the corresponding placeholder to obtain the prompt words of multiple data table samples.

[0062] In other embodiments, in order to improve the diversity of data and provide more references for subsequent prompt word generation, such as Figure 2 As shown in "Metadata Sampling and Segmentation", the metadata sample can also be segmented to obtain multiple metadata shards. In this embodiment, the specific implementation method of segmenting the metadata sample is not limited. In some embodiments, a single data feature of a single field can be segmented. For example, the value domain of a single field can be segmented into multiple subdomains. For another example, the sample value of a single field is segmented into multiple sample subsets, and so on. And / or, with the field as the segmentation granularity, the field information of some fields of the same data table is segmented into one metadata shard, and so on. The metadata sample of each data table can be segmented into multiple metadata shards.

[0063] After obtaining multiple metadata shards, the metadata shards belonging to the same data table can be combined to obtain multiple knowledge combinations (such as Figure 2 "Knowledge combination"). The combination is a concept in the field of mathematics. A knowledge combination may include a metadata shard, or may include some or all metadata shards belonging to the same data table. Specifically, any i metadata shards can be selected from the metadata shards belonging to the same data table as a combination. i=1,2,3…M. M represents the number of metadata shards into which the metadata sample of a data table is divided. Then the knowledge combination corresponding to the data table is indivual.

[0064] After obtaining multiple knowledge combinations, prompt words can be generated based on the multiple knowledge combinations. Among them, by segmenting the metadata samples of the same data table, combinations of different field information can be extracted from the same data table to form multi-dimensional knowledge fragments. These fragments can represent different features or perspectives, thereby increasing the diversity and richness of the data. Therefore, generating prompt words based on these knowledge combinations can improve the quality of the generated prompt words.

[0065] Specifically, in some embodiments, Figure 2 As shown in "Prompt Word Template", a preset prompt word template can be obtained. For any knowledge combination, the metadata contained in the knowledge combination can be written into the corresponding placeholder of the prompt word template according to the information represented by the placeholder in the prompt word template to generate the prompt word corresponding to the knowledge combination. Among them, one prompt word is generated for one knowledge combination. Generating prompt words according to the preset prompt word template can improve the efficiency and accuracy of prompt word generation.

[0066] After obtaining the prompt words for guiding the data generation model to generate the evaluation data, in step 103, the data generation model can be used to generate an evaluation data set for evaluating the question-and-answer application of the data source according to the prompt words.

[0067] In this embodiment, prompt words for guiding the data generation model to generate evaluation data are generated through the metadata of the data table in the data source, and the data generation model is used to generate an evaluation data set for evaluating the question and answer application of the data source according to the prompt words, thereby realizing the automatic generation of the evaluation data set. Compared with the manual annotation evaluation data set output method, the generation efficiency of the evaluation data set can be improved. On the other hand, since the metadata of the data table can reflect the structure and data characteristics of the data table, the prompt words generated according to the metadata of the data table can make the prompt words closely associated with the data table, and then the evaluation data generated according to the prompt words can be closely associated with the data table of the data source, which can ensure the validity of the generated evaluation data set.

[0068] In addition, through the data generation model, a large amount of evaluation data can be generated based on the prompt words, which can ensure that the evaluation data set contains a wide range of question types and structures, which helps to improve the representativeness and reliability of the evaluation results obtained by using the evaluation data to evaluate the question-and-answer application in the future. The evaluation data generated by the data generation model may cover some complex or edge cases that have not been encountered before, which helps to discover potential problems in the question-and-answer application.

[0069] In some embodiments, a piece of evaluation data may include: a natural statement describing the query requirement (i.e., a question), and may also include: a query statement corresponding to the question. The query statement is a query statement written in a programming language supported by the data source, such as an SQL query statement. For example, if the question is "Please give the performance indicators of server 001", the query statement is as follows: SELECT timestamp, / / timestamp cpu_usage, / / A performance indicator of the server, the utilization of the central processing unit (CPU) memory_usage, / / A performance indicator of the server, memory utilization disk_io, / / A performance indicator of the server, the input / output performance of the disk FROM server_performance / / Read data from the table named "server_performance" WHERE server_id = '001' / / The server ID is 001 AND timestamp >= DATE_SUB(CURRENT_TIMESTAMP, INTERVAL 1 DAY) / / Limit the result set to records within the past 24 hours ORDER BY timestamp ASC; / / Sort the results in ascending order by timestamp.

[0070] In this embodiment, the training samples of the data generation model include: question samples and query statements corresponding to the question samples, that is, the data generation model is trained with question samples and query statements corresponding to the question samples as training samples. Accordingly, the data generation model can be used to generate multiple target questions and query statements corresponding to the multiple target questions according to the aforementioned prompt words as an evaluation data set. Among them, one prompt word can generate at least one target question. Generally, one prompt word generates multiple target questions. Multiple refers to more than 2 (including 2). Since the mapping relationship between the target question and the query statement reflects the conversion logic between the natural language expression and the structured query language, the target question and the query statement are used as the subsequent evaluation data set to enhance the context understanding and semantic parsing capabilities of the question and answer application. On the other hand, when the performance of the question and answer application is evaluated later, the query statement generated by the question and answer application can be directly compared with the query statement in the evaluation data set to evaluate the performance of the question and answer application.

[0071] In some embodiments of the present application, in order to improve the quality of the generated evaluation data set, the evaluation data set generated in step 103 may also be quality checked. Since the evaluation data set includes multiple target questions and query statements corresponding to the multiple target questions, for any target question C, any target question C and the query statement corresponding to the target question C may be quality checked.

[0072] In some embodiments, the target question C can be parsed in natural language and key information in the target question C can be extracted; then, the key information in the target question C is matched with the structured elements in the query statement corresponding to the target question C; if the matches are consistent, it is determined that the target question C and the query statement corresponding to the target question C have passed the quality verification. This quality verification method can ensure the consistency between the target question and the query statement corresponding to it, which helps to improve the quality of the evaluation data.

[0073] In other embodiments, Figure 2 As shown in "Quality Verification - Query Statement Execution of Target Question", the query statement corresponding to target question C can be executed in the data source. If the query statement corresponding to target question C is executed successfully, it means that the query statement corresponding to target question C is consistent with the characteristics of the data table of the data source. Therefore, it can be determined that target question C and the query statement corresponding to target question C have passed the quality verification. Because only when the query statement completely conforms to the table structure, field name and data type of the data source can it be successfully executed, thus ensuring the accuracy of the query statement and helping to improve the quality of the evaluation data.

[0074] Or, if Figure 2 As shown in "Quality Verification—Relevance Verification", the relevance between the target question C and the query statement corresponding to the target question C can be determined. In this embodiment, the specific implementation method for determining the relevance between the target question C and the query statement corresponding to the target question C is not limited. In some embodiments, a pre-trained relevance determination model can be used to determine the relevance between the target question C and the query statement corresponding to the target question C. Among them, the relevance determination model can be a neural network model. The relevance determination model is trained using question samples and query statements corresponding to the question samples as training samples.

[0075] The output of the relevance determination model may be a relevance score between the question and the query statement corresponding to the question; or, the relevance determination model may be the relevance output by the classification model, which may be identification information indicating whether the question and the query statement corresponding to the question are relevant. For example, outputting "1" or "relevant" indicates that the question and the query statement corresponding to the question are relevant; outputting "0" or "not relevant" indicates that the question and the query statement corresponding to the question are not relevant, etc.

[0076] For an embodiment in which the relevance output by the relevance determination model is the relevance score between the question and the query statement corresponding to the question, when training the relevance determination model, the minimization of the first loss function is used as the training goal, and the model is trained with the question sample and the query statement corresponding to the question sample as the training sample to obtain the relevance determination model. The first loss function can be expressed as: the difference between the relevance score output by the model training and the pre-annotated relevance score. The pre-annotated relevance score can be 1 or 0, which is mainly determined by whether the question sample and the query statement corresponding to the question sample are related. If the question sample and the query statement corresponding to the question sample are related, the annotated relevance score is 1; if the question sample and the query statement corresponding to the question sample are irrelevant, the annotated relevance score is 0.

[0077] For an embodiment in which the relevance output by the relevance determination model is identification information of whether the question and the query statement corresponding to the question are related, when training the relevance determination model, the minimization of the second loss function is used as the training goal, and the model is trained with the question sample and the query statement corresponding to the question sample as the training sample to obtain the relevance determination model. The first loss function can be expressed as: the difference between the identification information output by the model training and the pre-labeled identification information. The pre-labeled identification information can be 1 or 0, which is mainly determined by whether the question sample and the query statement corresponding to the question sample are related. If the question sample and the query statement corresponding to the question sample are related, the labeled identification information can be 1; if the question sample and the query statement corresponding to the question sample are irrelevant, the labeled identification information can be 0.

[0078] Based on the above relevance determination model, the target question C and the query statement corresponding to the target question C can be input into the relevance determination model; and the relevance determination model is used to determine the relevance between the target question C and the query statement corresponding to the target question C according to the target question C and the query statement corresponding to the target question C. Among them, the relevance determination model that has been trained in advance is used to determine the relevance between the target question and the query statement corresponding to the target question. The relevance determination model can capture the deep-level features of the target question and the query statement corresponding to the target question, and therefore, the accuracy of determining the relevance between the target question and the query statement corresponding to the target question can be improved.

[0079] Alternatively, natural language analysis can be performed on the target question C, and key information in the target question C can be extracted; then, the key information in the target question C can be matched with the structured elements in the query statement corresponding to the target question C; and the ratio of the number of matched key information to the total number of key information can be calculated as the correlation between the target question C and the query statement corresponding to the target question C.

[0080] Further, if the correlation between the target question C and the query statement corresponding to the target question C is greater than or equal to the set correlation threshold, it is determined that the target question C and the query statement corresponding to the target question C have passed the quality verification. Among them, for the aforementioned embodiment in which the output of the correlation determination model is used as the correlation score between the target question and the query statement corresponding to the target question, and for the aforementioned embodiment in which the ratio of the number of matching key information to the total number of key information is used as the correlation between the target question C and the query statement corresponding to the target question C, the set correlation threshold may be greater than 0.5 and less than or equal to 1. For the aforementioned embodiment in which the output of the correlation determination model is used as an identifier of the correlation, this step does not need to be performed.

[0081] Among them, according to the correlation between the target question C and the query statement corresponding to the target question C, the quality of the target question C and the query statement corresponding to the target question C is verified, which can capture the deep logical relationship between the problem and the query statement, ensure that the query statement corresponding to the target question C and the target question C are semantically consistent, and avoid generating query statements that are grammatically correct but inconsistent with the question intent. Therefore, it helps to improve the quality of the evaluation data.

[0082] In other embodiments, the query statement corresponding to the target question C can be executed in the data source, and the correlation between the target question C and the query statement corresponding to the target question C can be determined; if the query statement corresponding to the target question C is successfully executed, and the correlation between the target question C and the query statement corresponding to the target question C is greater than or equal to the set correlation threshold, then it is determined that the query statement corresponding to the target question C and the query statement corresponding to the target question C have passed the quality verification. This quality inspection method can ensure that the syntax of the query statement is correct and the accuracy of the query statement is guaranteed on the one hand; on the other hand, it can capture the deep logical relationship between the question and the query statement, ensure that the semantics of the query statement corresponding to the target question C and the target question C are consistent, and avoid generating query statements that are grammatically correct but inconsistent with the question intent, thereby helping to improve the quality of the evaluation data.

[0083] In other embodiments, a piece of evaluation data may include: a natural sentence (i.e., a question) describing a query requirement, which is defined as a target question. Accordingly, the training sample of the data generation model includes: a question sample. That is, the data generation model is trained using the question sample as a training sample. Accordingly, the data generation model can be used to generate multiple target questions as an evaluation data set based on the aforementioned prompt words. Among them, one prompt word can generate at least one target question. Generally, one prompt word generates multiple target questions. Multiple refers to more than 2 (including 2).

[0084] In some embodiments of the present application, in order to improve the quality of the generated evaluation data set, the quality check may be performed on the evaluation data set generated in step 103. Since the evaluation data set includes multiple target problems, for any target problem C, the quality check may be performed on any target problem C.

[0085] In some embodiments, natural language analysis may be performed on the target question C, and key information in the target question C may be extracted; then, a query statement corresponding to the target question C may be generated based on the key information in the target question C. Then, the query statement may be executed in the data source to obtain a query result corresponding to the target question C. Then, the relevance between the target question C and the query result corresponding to the target question C may be determined.

[0086] In some embodiments, a pre-trained correlation determination model may be used to determine the correlation between the target question C and the query results corresponding to the target question C. Among them, the description of the correlation determination model can refer to the relevant content of the aforementioned embodiment, and will not be repeated here. Based on the above-mentioned correlation determination model, the target question C and the query results corresponding to the target question C can be input into the correlation determination model; and the correlation determination model is used to determine the correlation between the target question C and the query results corresponding to the target question based on the query results corresponding to the target question C. Among them, the correlation determination model that has been pre-trained is used to determine the correlation between the target question and the query results corresponding to the target question. The correlation determination model can capture the deep-level characteristics of the target question and the query results corresponding to the target question, and therefore, the accuracy of determining the correlation between the target question and the query results corresponding to the target question can be improved.

[0087] Furthermore, if the correlation between the target question C and the query result corresponding to the target question C is greater than or equal to the set correlation threshold, it is determined that the target question C passes the quality verification. According to the correlation between the target question C and the query result corresponding to the target question C, the quality verification of the target question C can capture the deep logical relationship between the question and the query result, and ensure that the target question C and the query result corresponding to the target question C are semantically consistent, which helps to improve the quality of the evaluation data.

[0088] The quality verification method for the evaluation data set shown in the above embodiment is only an example and does not constitute a limitation. The same method can be used to complete the quality verification of each evaluation data (including: target questions and query statements corresponding to the target questions). Further, from the evaluation data set, the target evaluation data that has passed the quality verification is obtained as the evaluation data for evaluating the question and answer application of the data source. Using the target evaluation data that has passed the quality verification as the final evaluation data for evaluating the question and answer application can improve the quality of the evaluation data and help improve the reliability of subsequent evaluations of the question and answer application. In some embodiments, such as Figure 2 As shown in "Evaluation Data Storage", the target evaluation data that has passed the quality verification can be stored for use in subsequent evaluation of the question-and-answer application.

[0089] It is worth noting that the data generation method provided in the embodiment of the present application can be deployed on any computing device. Optionally, the data generation method provided in the embodiment of the present application can also be deployed on a cloud server as a software as a service (SaaS) application. For a cloud server deployed with the SaaS application, the steps in the above evaluation data generation method can be executed in response to a request to call the target service. The specific implementation method is as follows Figure 3 As shown, the method is applicable to a cloud server and mainly includes: 301. In response to a request to call a target service, determine a processing resource corresponding to the target service. The target service refers to a service that provides a method for generating evaluation data.

[0090] 302. Using the processing resources corresponding to the target service, obtain metadata of the data table in the data source.

[0091] 303. Generate prompt words based on metadata using processing resources corresponding to the target service. The prompt words are used to guide the pre-trained data generation model to generate evaluation data.

[0092] 304. Using the processing resources corresponding to the target service and the data generation model, an evaluation data set for evaluating the question-and-answer application of the data source is generated according to the prompt words.

[0093] In this embodiment, the target service refers to a service that provides a method for generating evaluation data. The processing resources corresponding to the target service refer to the processing resources required to execute the above-mentioned method for generating evaluation data, including but not limited to: processor resources, memory resources and IO resources.

[0094] The method for generating evaluation data provided in this embodiment can be deployed on a cloud server to provide users with a service for generating evaluation data of a target application, namely, a target service. The user may be a service provider of the target application, or a user or client of the target application. Optionally, the cloud service end may provide an application programming interface (Application ProgramInterface, API) to the user. The service requester (ie, the user) may call the API to call the target service. Accordingly, the request to call the target service is implemented as a call event generated by calling the API. The service requester (ie, the user) may also call the target service through remote procedure call (Remote Procedure Call, RPC) or remote direct memory access (Remote Direct Memory Access, RDMA) technology.

[0095] For the cloud server, in response to the request to call the target service, the processing resources corresponding to the target service can be determined; and the processing resources corresponding to the target service can be used to perform steps 302 to 304 to automatically generate evaluation data. For the specific implementation of steps 302 to 304, please refer to the above Figure 1 The relevant content will not be repeated here.

[0096] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 and 102 can be device A; for another example, the execution subject of step 101 can be device A, and the execution subject of step 102 can be device B; and so on.

[0097] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel, and the serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel.

[0098] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the aforementioned evaluation data generating method.

[0099] The computer-readable storage medium includes volatile or non-volatile or a combination thereof, and may be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media.

[0100] The embodiment of the present application also provides a computer program product, including a computer program. When the computer program is executed by one or more processors, the one or more processors are caused to execute the steps in the aforementioned evaluation data generating method.

[0101] In the embodiments of the present application, the specific implementation form of the computer program product is not limited. In some embodiments, the computer program product can be implemented as an application (Application, APP), a small program, a computer-side client, a program module, a plug-in, an installation package, a software development kit (Software Development Kit, SDK), an image file of a disc (such as an ISO file), a plug-in or software as a service (Software as a Service, SaaS) software, etc., but is not limited thereto.

[0102] It should be understood that each process or a combination of multiple processes in the above method process can be implemented by a computer program or instruction. In addition, these computer programs or instructions can be applied to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device, so that the processor of the general-purpose computer, the special-purpose computer, the embedded processor or other programmable data processing device can be implemented as a device to implement the corresponding functions in the above method embodiments.

[0103] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the electronic device includes: a memory 40a and a processor 40b. The memory 40a is used to store computer programs and can be configured to store various other data to support operations on a computing platform. Examples of these data include instructions, data structures, contact data, phone book data, messages, pictures, videos, etc. for any application or method operating on the electronic device.

[0104] The processor 40b is coupled to the memory 40a and is used to execute the computer program to execute the steps in the evaluation data generation method provided in the above embodiments. The specific implementation of each step can be found in the relevant description of the above embodiments, which will not be repeated here.

[0105] In some optional embodiments, such as Figure 4 As shown, the electronic device may also include optional components such as a communication component 40c, a power component 40d, a display component 40e and an audio component 40f. Figure 4 The components are shown schematically only, and do not mean that the electronic device must include Figure 4 The components shown do not necessarily mean that the electronic device can only include Figure 4 Components shown.

[0106] in addition, Figure 4 The components in the dashed box are optional components, not mandatory components, and may depend on the product form of the electronic device. The electronic device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a mobile phone or an Internet of Things device; it can also be a traditional server, a cloud server or a server cluster.

[0107] In an embodiment of the present application, the memory is used to store a computer program and can be configured to store various other data to support operations on the device where it is located. Among them, the processor can execute the computer program stored in the memory to implement the corresponding control logic. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), electrically erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0108] In the embodiment of the present application, the processor may be any hardware processing device that can execute the logic of the above method. Optionally, the processor may be a central processing unit (CPU), a graphics processing unit (GPU) or a microcontroller unit (MCU); it may also be a field programmable gate array (FPGA), a programmable array logic device (PAL), a general array logic device (GAL), a complex programmable logic device (CPLD) or other programmable devices; or it may be an advanced reduced instruction set (RISC) processor (Advanced RISC Machines, ARM) or a system on chip (System on Chip, SoC), etc., but not limited thereto.

[0109] In an embodiment of the present application, the communication component is configured to facilitate wired or wireless communication between the device in which it is located and other devices. The device in which the communication component is located can access a wireless network based on a communication standard, such as 2G or 3G, 4G, 5G or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel.

[0110] In an embodiment of the present application, the display component may include a liquid crystal display (LCD) and a touch panel (TP). If the display component includes a touch panel, the display component may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0111] In an embodiment of the present application, a power supply component is configured to provide power to various components of the device in which it is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.

[0112] In an embodiment of the present application, the audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal. For example, for a device with a language interaction function, voice interaction with a user may be achieved through an audio component.

[0113] It should be noted that the descriptions such as “first” and “second” in this article are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit “first” and “second” to different types.

[0114] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the above elements.

[0115] The above contents are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for generating evaluation data, characterized in that: include: Get the metadata of the data table in the data source; generating prompt words according to the metadata; The prompt word is used to guide the pre-trained data generation model to generate evaluation data; The data generation model is used to generate an evaluation data set for evaluating the question-and-answer application of the data source according to the prompt word.

2. The method according to claim 1, characterized in that Generating a prompt word according to the metadata includes: Sampling the metadata to obtain a metadata sample; Slicing the metadata sample to obtain a plurality of metadata slices; Combine metadata shards belonging to the same data table to obtain multiple knowledge combinations; The prompt word is generated according to the plurality of knowledge combinations.

3. The method according to claim 2, characterized in that The step of sampling the metadata to obtain a metadata sample includes: Determine the target sampling ratio according to the amount of evaluation data in the evaluation data set to be generated and the amount of data in the data table; The metadata is sampled according to the target sampling ratio to obtain the metadata sample.

4. The method according to claim 2, characterized in that The step of generating the prompt word according to the plurality of knowledge combinations includes: Get the preset prompt word template; For any knowledge combination, according to the information represented by the placeholder in the prompt word template, the metadata included in the any knowledge combination is written into the corresponding placeholder of the prompt word template to generate the prompt word corresponding to the any knowledge combination.

5. The method according to claim 1, characterized in that The data generation model is obtained by training with question samples and query sentences corresponding to the question samples as training samples; The step of using the data generation model to generate an evaluation data set for evaluating the question-and-answer application of the data source according to the prompt word includes: The data generation model is used to generate multiple target questions and query statements corresponding to the multiple target questions according to the prompt words, as the evaluation data set.

6. The method according to any one of claims 1 to 5, characterized in that: Also includes: Performing quality verification on the evaluation data set; Target evaluation data that has passed quality verification is obtained from the evaluation data set as evaluation data for evaluating the question-and-answer application of the data source.

7. The method according to claim 6, characterized in that The evaluation data set includes a plurality of target questions and query statements corresponding to the plurality of target questions; the quality verification of the evaluation data set includes: For any target question, executing a query statement corresponding to any target question in the data source; if the query statement corresponding to any target question is successfully executed, determining that any target question and the query statement corresponding to any target question pass the quality verification; or, Determine the relevance between any one of the target questions and the query statement corresponding to the any one of the target questions; if the relevance is greater than or equal to a set relevance threshold, determine that any one of the target questions and the query statement corresponding to the any one of the target questions pass the quality verification; or, Execute a query statement corresponding to any target question in the data source, and determine the correlation between any target question and the query statement corresponding to any target question; if the query statement corresponding to any target question is executed successfully, and the correlation is greater than or equal to a set correlation threshold, then determine that any target question and the query statement corresponding to any target question pass the quality verification.

8. The method according to claim 7, characterized in that The determining the relevance between any one of the target questions and the query statement corresponding to the any one of the target questions includes: Inputting any target question and the query sentence corresponding to any target question into a pre-trained relevance determination model; the relevance determination model is trained using the question sample and the query sentence corresponding to the question sample as training samples; The relevance is determined by using the relevance determination model according to any one of the target questions and a query statement corresponding to the any one of the target questions.

9. The method according to any one of claims 1 to 5, characterized in that: The step of obtaining metadata of a data table in a data source includes: Obtaining basic information of the data table from the system table of the data source as the metadata; and / or, The field information of the data table is obtained as the metadata.

10. The method according to claim 9, characterized in that The obtaining of the field information of the data table includes: Obtaining structural information of fields of the data table from the system table of the data source; Obtaining data features of fields of the data table from the data source; The structural information of the field and the data characteristics of the field are determined as the field information.

11. The method according to claim 10, characterized in that The step of obtaining data features of fields of the data table from the data source includes: Acquire, from the data source, a value domain of a field of the data table as a data feature of the field of the data table; and / or, For any field of the data table, sampling the value of any field contained in the data table according to a set first sampling ratio to obtain a first sample value of the any field as a data feature of the any field; and / or, Determine, according to the value of any one of the fields, a statistical feature of the any one of the fields as a data feature of the any one of the fields; and / or, According to the value of any one of the fields, determining an outlier of the any one of the fields as a data feature of the any one of the fields; and / or, From the values ​​of any one of the fields, a target value that satisfies a set condition is determined as a data feature of any one of the fields.

12. The method according to claim 11, characterized in that The obtaining, from the data source, a value range of a field of the data table includes: For any field of the data table, when the amount of data in the data source is greater than or equal to a set data amount threshold, sampling the value of the any field contained in the data source according to a set second sampling ratio to obtain a second sample value of the any field; determining the value range of the any field according to the second sample value of the any field; or, When the data volume of the data source is less than a set data volume threshold, the value of any one of the fields is obtained from the data source; and the value range of any one of the fields is determined according to the value of any one of the fields.

13. A method for generating evaluation data, characterized in that: include: In response to a request to invoke a target service, determining a processing resource corresponding to the target service; The target service refers to a service that provides a method for generating evaluation data; The processing resources are used to perform the steps of the method according to any one of claims 1 to 12.

14. An electronic device, characterized in that: include: A memory and a processor; wherein the memory is used to store a computer program; The processor is coupled to the memory and configured to execute the computer program to perform the steps of the method according to any one of claims 1 to 13.

15. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the method according to any one of claims 1 to 13.

16. A computer program product, characterized in that The method comprises a computer program which, when executed by one or more processors, causes the one or more processors to execute the steps of the method according to any one of claims 1 to 13.

Images