Evaluation data set construction method and device, computer equipment and readable storage medium

Abstract

The embodiment of the invention provides an evaluation data set construction method and device, computer equipment and a readable storage medium. The method comprises the following steps: receiving a data set acquisition request sent by a target client, and reading a candidate data set according to the data set acquisition request; labeling each piece of original data according to a preset attribute labeling rule to obtain candidate labeling data, each candidate labeling data being associated with a plurality of attribute labels; on the basis of the multiple attribute tags, the multiple candidate annotation data are subjected to duplicate removal processing, and multiple target annotation data are obtained; constructing at least one piece of problem statement data for each piece of target annotation data, and performing question element updating on each piece of target annotation data based on the at least one piece of problem statement data to obtain updated annotation data; and constructing an evaluation data set according to the plurality of updated annotation data, and sending the evaluation data set to the target client. In this way, the applicability of the reconstructed evaluation data set can be improved while redundant data in the data set can be effectively removed.

Description

Technical Field

[0001] The present application relates to the field of model evaluation technology, and in particular to a method, apparatus, computer equipment, and readable storage medium for constructing an evaluation dataset. Background Art

[0002] Model evaluation is the process of evaluating the performance of large language models (e.g., dialogue, reasoning, code generation, and other capabilities) through systematic methods and metrics. Building an appropriate dataset is a key foundation for this evaluation. Carefully designed evaluation datasets can cover capabilities at various levels, from basic language comprehension to complex logical reasoning, thereby comprehensively examining the model's generalization capabilities and knowledge application level.

[0003] In related technologies, multiple user-selected datasets are typically concatenated to construct a single evaluation dataset for model evaluation. On the one hand, multiple datasets often contain duplicate data, and directly combining them results in a high level of redundant data in the evaluation dataset. On the other hand, large language models often struggle to adapt to evaluation using directly combined evaluation datasets, resulting in a low applicability of the evaluation dataset. Summary of the Invention

[0004] The present application proposes a method, apparatus, computer device and readable storage medium for constructing an evaluation dataset, which can effectively remove redundant data in the dataset while improving the applicability of the reconstructed evaluation dataset.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a method for constructing an evaluation dataset, the method comprising:

[0006] Receive a dataset acquisition request sent by a target client, and read a candidate dataset according to the dataset acquisition request, wherein the candidate dataset includes a plurality of original data;

[0007] Annotate each original data according to a preset attribute annotation rule to obtain each candidate annotated data after the annotation process, each candidate annotated data being associated with a plurality of attribute labels, wherein the plurality of attribute labels at least include a question attribute label and an option attribute label;

[0008] Based on the multiple attribute labels associated with each candidate labeled data, performing deduplication processing on the multiple candidate labeled data to obtain multiple target labeled data;

[0009] For each target annotated data, construct at least one question statement data according to the question attribute information corresponding to the question attribute tag and the option attribute information corresponding to each option attribute tag, and perform a topic element update on each target annotated data based on the at least one question statement data to obtain updated annotated data;

[0010] An evaluation data set is constructed according to the plurality of updated annotation data, and the evaluation data set is sent to the target client.

[0011] Accordingly, a second aspect of the embodiments of the present application provides a device for constructing an evaluation dataset, the device comprising:

[0012] A receiving module, configured to receive a dataset acquisition request sent by a target client, and read a candidate dataset according to the dataset acquisition request, wherein the candidate dataset includes a plurality of original data;

[0013] A labeling module is used to label each raw data according to a preset attribute labeling rule to obtain each candidate labeled data after labeling, each candidate labeled data being associated with a plurality of attribute labels, wherein the plurality of attribute labels include at least a question attribute label and an option attribute label;

[0014] a deduplication module, configured to perform deduplication processing on the plurality of candidate labeled data based on the plurality of attribute labels associated with each candidate labeled data, to obtain a plurality of target labeled data;

[0015] a construction module configured to construct, for each target annotated data, at least one question statement data according to the question attribute information corresponding to the question attribute tag and the option attribute information corresponding to each option attribute tag, and perform a topic element update on each target annotated data based on the at least one question statement data to obtain updated annotated data;

[0016] The sending module is used to construct an evaluation data set according to the multiple updated annotation data, and send the evaluation data set to the target client.

[0017] In some implementations, the multiple attribute labels further include a category attribute label, an answer attribute label, and a background attribute label, and the deduplication module is further configured to:

[0018] Based on at least one of the background attribute label, the question attribute label, the answer attribute label, and the option attribute label of each candidate annotation data, combined with the category attribute label, global attribute label data is obtained;

[0019] Performing hash calculation on the global attribute tag data to obtain the corresponding data fingerprint;

[0020] Compare multiple data fingerprints to obtain comparison results;

[0021] When the comparison result indicates that there are at least two target data fingerprints with the same fingerprint among the multiple data fingerprints, duplicate removal is performed on the candidate labeled data corresponding to the at least two target data fingerprints to obtain multiple target labeled data.

[0022] In some implementations, the plurality of attribute tags further include an answer attribute tag, and the building module is further configured to:

[0023] For each target annotated data, determining a corresponding question-and-answer mode based on the question attribute information corresponding to the question attribute label, the option attribute information corresponding to each option attribute label, and the answer attribute information corresponding to the answer attribute label, wherein the question-and-answer mode includes at least one of an option question-and-answer mode, a statement scoring mode, and a judgment scoring mode;

[0024] Based on the question-answering mode, combining the question attribute information with the corresponding option attribute information to generate corresponding initial question statement data;

[0025] The initial question statement data is concatenated with the corresponding answer attribute information to obtain at least one piece of question statement data.

[0026] In some embodiments, the evaluation data set construction apparatus further includes a matching module for:

[0027] Dividing the plurality of updated annotated data into an evaluation subset and an example subset according to a preset ratio, and matching each piece of evaluation data in the evaluation subset with the example subset according to the category attribute information corresponding to the category attribute label of the evaluation data and the background attribute information corresponding to the background attribute label, to obtain at least one target example data;

[0028] Establishing a mapping relationship between each piece of evaluation data and the corresponding at least one piece of target example data;

[0029] Then, constructing the evaluation data set based on the multiple updated annotation data includes:

[0030] An evaluation data set is constructed according to the plurality of evaluation data.

[0031] In some embodiments, the matching module is further configured to:

[0032] splicing the category attribute information and the background attribute information corresponding to the evaluation data to obtain a query entry;

[0033] Matching the query item with multiple pieces of example data in the example subset to obtain corresponding multiple matching similarities;

[0034] Based on the magnitude relationship of the multiple matching similarities, at least one target example data corresponding to the evaluation data is determined from the multiple example data included in the example subset.

[0035] In some embodiments, the annotation module is further configured to:

[0036] Perform attribute recognition on each raw data according to the preset attribute labeling rules to obtain the recognition result;

[0037] Each original data is labeled according to the recognition result to obtain each candidate labeled data after the labeling process.

[0038] In some implementations, the sending module is further configured to:

[0039] Determining a corresponding knowledge dimension density index based on the number of knowledge points contained in each of the updated annotation data;

[0040] Calculating the option similarity between any two option sub-data according to the plurality of option sub-data included in the option attribute information corresponding to each updated annotation data, and obtaining a corresponding interference index according to the sum of the plurality of option similarities corresponding to the plurality of option sub-data;

[0041] Determining an evaluation level of each corresponding updated labeled data according to the knowledge dimension density index and the interference index, and marking each updated labeled data based on the evaluation level to obtain target labeled data;

[0042] Construct an evaluation dataset based on multiple target labeled data.

[0043] Correspondingly, the third aspect of the embodiments of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the evaluation data set construction method of any one of the embodiments of the first aspect of the present application.

[0044] Correspondingly, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the evaluation data set construction method of any one of the embodiments of the first aspect of the present application.

[0045] The present application receives a data set acquisition request sent by a target client, and reads a candidate data set according to the data set acquisition request, wherein the candidate data set includes multiple original data; each original data is annotated according to a preset attribute annotation rule to obtain each candidate annotated data after the annotation processing, each candidate annotated data is associated with multiple attribute labels, and the multiple attribute labels include at least a question attribute label and an option attribute label; based on the multiple attribute labels associated with each candidate annotated data, multiple candidate annotated data are deduplicated to obtain multiple target annotated data; for each target annotated data, at least one question statement data is constructed according to the question attribute information corresponding to the included question attribute label and the option attribute information corresponding to each option attribute label, and the subject element of each target annotated data is updated based on the at least one question statement data to obtain updated annotated data; an evaluation data set is constructed according to the multiple updated annotated data, and the evaluation data set is sent to the target client. In this way, the structured management of the evaluation data can be achieved by performing fine-grained attribute division and labeling on the data. This structured processing method can accurately identify and eliminate duplicate data on the core classification dimension between different data sets, and solves the redundancy problem caused by directly combining multiple data sets while retaining the data diversity characteristics. On the other hand, this application can dynamically construct diversified question statements based on attribute information and update the question elements, so that the evaluation dataset not only retains the core evaluation elements, but also generates variant data adapted to different evaluation scenarios, thereby making the structure and content of each data more consistent with the evaluation requirements of the large language model, effectively improving the coverage breadth and test validity of the evaluation dataset for the large language model capability dimension. In summary, this application can effectively remove redundant data in the dataset while improving the applicability of the reconstructed evaluation dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a schematic diagram of the architecture of the evaluation data set construction system provided in an embodiment of the present application;

[0047] Figure 2 This is a flowchart of the method for constructing an evaluation data set provided in an embodiment of the present application;

[0048] Figure 3 This is a schematic diagram of the functional modules of the evaluation data set construction device provided in an embodiment of the present application;

[0049] Figure 4 This is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0051] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0053] Model evaluation is the process of evaluating the performance of large language models (e.g., dialogue, reasoning, code generation, and other capabilities) through systematic methods and metrics. Building an appropriate dataset is a key foundation for this evaluation. Carefully designed evaluation datasets can cover capabilities at various levels, from basic language comprehension to complex logical reasoning, thereby comprehensively examining the model's generalization capabilities and knowledge application level.

[0054] In related technologies, multiple user-selected datasets are typically concatenated to construct a single evaluation dataset for model evaluation. On the one hand, multiple datasets often contain duplicate data, and directly combining them results in a high level of redundant data in the evaluation dataset. On the other hand, large language models often struggle to adapt to evaluation using directly combined evaluation datasets, resulting in a low applicability of the evaluation dataset.

[0055] Based on this, the embodiments of the present application provide a method, apparatus, computer device, and readable storage medium for constructing an evaluation dataset, which can effectively remove redundant data in the dataset while improving the applicability of the reconstructed evaluation dataset.

[0056] The evaluation dataset construction method, apparatus, computer device, and readable storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the evaluation dataset construction system in the embodiments of the present application is described.

[0057] Please refer to Figure 1 In some implementations, an embodiment of the present application provides an evaluation data set construction system, including a terminal 11 and a server 12 .

[0058] For example, the terminal 11 can be a graphics workstation, personal computer, laptop, tablet computer, or smart phone used in daily life. When a user on the target client (running on the terminal 11) needs to obtain an evaluation dataset to evaluate the large language model, the terminal 11 sends this request to the server 12.

[0059] Furthermore, the server 12 can be a physical server, a cloud server, or a server cluster. After receiving the dataset acquisition request sent by the terminal 11, the server 12 can obtain the corresponding candidate dataset from a public database or other channel, process the candidate dataset, construct an evaluation dataset, and send it back to the terminal 11. The terminal 11 receives the evaluation dataset and displays it to the user to meet the user's demand for an adapted large-scale model evaluation dataset.

[0060] The evaluation data set construction method in the embodiments of the present application can be illustrated by the following examples.

[0061] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0062] In the embodiment of the present application, the evaluation data set construction device will be described from the perspective of the evaluation data set construction device, which can be integrated into a computer device. Figure 2 , Figure 2 This is a flowchart of the steps of the evaluation dataset construction method provided in an embodiment of the present application. In this embodiment of the present application, the evaluation dataset construction device is specifically integrated into a terminal or server as an example. When the processor on the terminal or server executes the program instructions corresponding to the evaluation dataset construction method, the specific process is as follows:

[0063] Step 101: Receive a dataset acquisition request sent by a target client, and read a candidate dataset according to the dataset acquisition request, where the candidate dataset includes a plurality of original data.

[0064] In some implementations, in order to construct an evaluation dataset that meets the requirements, user evaluation requirements can be received and matched with preset data resources, so as to facilitate flexible calling and standardized preprocessing of the dataset during the large model evaluation process.

[0065] Among them, the target client can be a user interaction terminal deployed in a large language model evaluation system.

[0066] The dataset acquisition request can be a structured data call instruction containing the target assessment capability type code (such as R2 for mathematical reasoning) and can also specify a specific candidate dataset. The dataset acquisition request can be a metadata package in JSON format, which is used to trigger index matching of the candidate dataset.

[0067] Among them, the candidate dataset can be a dataset that has been selected based on the dataset acquisition request and has completed category attribute annotation and is stored in a distributed file system, or a dataset stored in a public database. Each candidate dataset meets the large model evaluation benchmark.

[0068] The original data may be an atomic evaluation unit in the candidate data set.

[0069] In some implementations, when a dataset acquisition request specifies a specific candidate dataset, the system can first verify the user's access rights to the corresponding candidate dataset using an access control list. If verification is successful, the dataset acquisition request is read. For example, if a user explicitly specifies in a dataset acquisition request that they wish to use the AGIEval and QuAIL raw datasets to construct an evaluation dataset, the system can obtain the required raw datasets from open source dataset platforms, academic resource repositories, professional evaluation platforms, cloud service platforms, or self-built databases.

[0070] In some embodiments, when a dataset acquisition request does not specify a specific candidate dataset, a deep semantic understanding model can be used to parse the dataset acquisition request into a structured query object. For example, when the dataset acquisition request is "requires multiple-choice questions of mathematical reasoning, with a difficulty level of medium to high", the deep semantic understanding model first parses the request and converts it into a structured query object, which contains key information such as the field of "mathematical reasoning" and the difficulty level of "medium to high". Afterwards, the system performs keyword matching and semantic similarity calculations in the semantic meta-information library of the dataset to filter out datasets that meet the "mathematical reasoning" field, and further filters out the parts with a difficulty level of "medium to high" based on the difficulty label. At the same time, the system performs a quality assessment on the filtered datasets, including whether the number of questions is sufficient, whether the question diversity is rich, and the accuracy of the answers, so as to obtain candidate datasets.

[0071] Furthermore, the user can also set the number of datasets through the dataset acquisition request, for example, one candidate dataset, three candidate datasets, ten candidate datasets, and so on. If the candidate datasets initially screened by the system are larger than the number of datasets set by the user, then the system can select the dataset with the largest download volume as the final candidate dataset based on the download volume of these datasets. Alternatively, the system can randomly select candidate datasets equal to the number of datasets to meet the user's needs. This embodiment of the application does not impose any specific restrictions on this.

[0072] For example, the original data may include questions, options and answers. For example, the question may be: "A certain e-commerce platform offers a 50 yuan discount for purchases over 300 yuan. The total price of the shopping cart items is 580 yuan. How much should I actually pay?" The options may be "A. 510 yuan, B. 520 yuan, C. 530 yuan, D. 540 yuan", and the answer may be: "530 yuan".

[0073] By establishing a standardized data request response mechanism, the call delay caused by the physical isolation of multiple data sets can be eliminated, the precise routing of evaluation requirements and the dynamic loading of data resources can be achieved, and the efficiency of data set processing can be improved.

[0074] Step 102 : annotate each original data according to preset attribute annotation rules to obtain each candidate annotated data after the annotation process. Each candidate annotated data is associated with multiple attribute labels, and the multiple attribute labels include at least a question attribute label and an option attribute label.

[0075] In some embodiments, in order to effectively solve the problem of multi-task adaptation difficulties caused by the lack of attributes in the original data set, unstructured evaluation data can be converted into a standard metadata format through structured attribute annotation. This can provide standardized basic data support for the construction of subsequent data sets, thereby improving the quality of the constructed evaluation data sets.

[0076] The preset attribute labeling rules may be a set of predefined standards and guidelines followed when labeling the original data, and a set of rules for extracting and labeling multiple attribute labels from the original data.

[0077] The candidate labeled data may be candidate data with corresponding attribute labels added.

[0078] The attribute label can be a metadata identifier that represents the semantic features of the data sample. For example, the attribute label can be a category attribute label, a background attribute label, a question attribute label, an option attribute label, an answer attribute label, etc.

[0079] The question attribute label may be a metadata field used to identify the questioning intent of the data core.

[0080] Among them, the option attribute tag can be a metadata field for structured storage of alternative answers to objective questions, and the corresponding option attribute information includes options for objective questions such as single-choice questions and multiple-choice questions.

[0081] For example, consider a raw data point from the AGIEval dataset. Consider a raw data point describing comments about the outcome of a game, with the question asking to infer the winning team based on these comments. First, based on pre-set attribute labeling rules, it can be determined that this question involves logical analysis, so a category attribute label, "Logical Reasoning," can be added. Similarly, a context attribute label can be added to the audience comments regarding the results of the five-team match. A question attribute label can be added to the question "Infer which team won the championship based on the audience comments." Option attribute labels can also be added to all listed answer options, such as "A, Team One," "B, Team Two," "C, Team Three," and "D, Team Four." Finally, an answer attribute label can be added to the final answer, "A, Team One," and so on. Ultimately, the resulting candidate labeled data not only contains the original question and option information but also includes multiple attribute labels that help the large language model understand the context and type of the task, enabling users to more accurately evaluate the large language model's performance on specific tasks.

[0082] In some implementations, the attribute labels may also include cognitive complexity labels, difficulty level labels, and the like.

[0083] Through the above methods, the information in the candidate data set can be systematically organized and managed, the data's structuring and searchability can be improved, the evaluation data can be more accurately matched to the model evaluation requirements, and the comprehensiveness and effectiveness of the evaluation can be ensured, thereby improving the efficiency and accuracy of large model evaluation.

[0084] In some implementations, in order to provide standardized input for subsequent data deduplication and evaluation format conversion, the original data can be attributed and annotated using automated semantic parsing technology to dynamically bind the extracted semantic attributes to the original data, better meeting the needs of specific application scenarios (such as model evaluation) and ensuring that the data can accurately and efficiently support the execution of related tasks. For example, step 102 may include:

[0085] (102.1) performing attribute recognition on each raw data according to a preset attribute labeling rule to obtain a recognition result;

[0086] (102.2) Each original data is labeled according to the recognition result to obtain each candidate labeled data after the labeling process.

[0087] The recognition result may be a structured attribute value parsed from the original data, for example, identifying that "computing champion team" belongs to the data corresponding to the problem attribute label.

[0088] In some implementations, a dependency parser (e.g., Stanford CoreNLP) can be used to extract sentence trunks, identify core predicates and arguments, and then accurately identify attributes for each raw data item using the identified core predicates and arguments. For example, a core predicate (e.g., "calculate") and an argument (e.g., "champion team") can be identified to determine the question attribute information corresponding to the question attribute label (Q).

[0089] Furthermore, based on the regular expression library, preset patterns can be matched to perform attribute recognition on each piece of raw data. For example, the text block after "option:" is automatically classified as an option attribute tag, and so on.

[0090] For example, to better understand how to use the dependency parser and regular expression library to identify and label attributes of raw data, we can use a specific example to illustrate this process. For example, if the raw data is a sample of raw data from the AGIEval dataset, the specific example is as follows:

[0091] "Five teams participated in the competition. Spectators made the following comments on the results: (1) Team A or Team B was the winner; (2) Team C and Team D were the winners; (3) Team D won the championship; (4) Team A was not the winner."

[0092] Question: "Based on the above comments, which team won the championship?"

[0093] Options: AA team, BD team, CC team, DB team.

[0094] Correct answer: Team CC.

[0095] Based on the above raw data, the sentence trunk is extracted using a dependency parser. First, the question "Based on the above comments, which team won the championship?" is input into the dependency parser. The dependency parser parses the sentence structure and identifies the core predicate (such as "win") and its arguments (such as "champion team"). For example, in this question, "win" is the core predicate, and "which team" is the argument. Then, based on the core predicate and argument, the question attribute information corresponding to the question attribute label (Q) can be determined as "Which team won the championship", and thus it can be marked as the question attribute label (Q).

[0096] Furthermore, regular expression patterns can be set to identify specific attributes. Each attribute tag can have multiple regular expression patterns. For example, the option attribute tag can correspond to "Option:", "Select the correct answer:", and so on. For example, a pattern can be defined to match all text blocks after "Option:" (i.e., option attribute information) and annotate them as the option attribute tag (O). Similarly, attribute information corresponding to attribute tags such as the answer attribute tag (A) can also be identified and annotated using a set regular expression pattern.

[0097] In some implementations, the raw data content can be analyzed based on keyword matching to determine if the question belongs to the logical reasoning type. For example, when the keywords are "analysis" or "inference," the corresponding data category attribute label (C) can be marked as "logical reasoning." Furthermore, the category attribute label can also be manually annotated by technicians.

[0098] In some implementations, the background attribute information can usually be directly extracted from the previous text. Therefore, the corresponding data can be directly labeled with the corresponding background attribute label (B). This embodiment of the present application does not impose any specific restrictions on this.

[0099] Through the above methods, the structure and parseability of the data can be enhanced. At the same time, the constructed evaluation dataset can be more flexibly responded to diverse evaluation needs, ensuring the quality and applicability of the dataset.

[0100] Step 103 : Based on the multiple attribute labels associated with each candidate annotation data, duplicate removal is performed on the multiple candidate annotation data to obtain multiple target annotation data.

[0101] In some implementations, in order to avoid evaluation bias and resource waste due to data redundancy, the data read and labeled from multiple candidate data sets can be deduplicated, thereby improving the quality of the data set used for model evaluation.

[0102] The target labeling data may be a unique data unit retained after deduplication processing, that is, there is no other data that is duplicated with the target labeling data.

[0103] In some embodiments, a globally unique category attribute label (C) can be constructed to add a unique category label to each piece of data in each original dataset, thereby reconstructing the dataset based on the category attribute to ensure that the category labels of any two pieces of data are non-duplicate. For example, for candidate labeled data 11 in candidate dataset 1, which corresponds to category attribute label A, then candidate labeled data 21 in candidate dataset 2, which corresponds to category attribute label A, when adding category attribute label A to candidate labeled data 21, it is detected that the content of candidate labeled data 11 and candidate labeled data 21 is the same, so candidate labeled data 21 can be discarded. Similarly, other candidate labeled data can also be deduplicated using the above method.

[0104] In some implementations, a hash value-based method may be used to convert each candidate annotation data into a unique hash value, and duplicate data may be quickly detected and deleted by comparing hash values ​​to achieve rapid deduplication of multiple candidate annotation data.

[0105] Furthermore, it is possible to combine the deep learning model to train a binary classifier specifically for identifying duplicate data, which can intelligently determine whether the data is duplicated and then quickly deduplicate multiple candidate labeled data.

[0106] Through the above methods, duplicate data can be effectively eliminated and the uniqueness of each data can be ensured, which not only improves the quality and representativeness of the data set, but also avoids subsequent evaluation bias and resource waste caused by data redundancy, thereby improving user satisfaction.

[0107] In some embodiments, in order to provide high-quality data input without redundancy for the evaluation task, deduplication processing of multiple candidate annotation data can be performed based on hash calculation to ensure that the candidate annotation data from different sources are uniquely constrained at the logical level, thereby ensuring the purity and representativeness of the data set. For example, step 103 may include:

[0108] (103.1) combining at least one of the background attribute label, the question attribute label, the answer attribute label, and the option attribute label of each candidate annotation data with the category attribute label to obtain global attribute label data;

[0109] (103.2) Perform hash calculation on the global attribute tag data to obtain the corresponding data fingerprint;

[0110] (103.3) Compare multiple data fingerprints to obtain a comparison result;

[0111] (103.4) When the comparison result indicates that there are at least two target data fingerprints that are consistent among the multiple data fingerprints, deduplication processing is performed on the candidate labeled data corresponding to the at least two target data fingerprints to obtain multiple target labeled data.

[0112] Among them, the background attribute tag can be a metadata field used to identify the context of the candidate annotation data, that is, used to mark the background information necessary for answering questions, such as the original text in reading comprehension, cases in legal examinations, etc.

[0113] The answer attribute tag may be an independent data unit used to mark the standard answer.

[0114] The category attribute label may be a classification identifier used to characterize the domain to which the data belongs.

[0115] Among them, the global attribute label data can be a composite metadata package generated by using vector splicing technology, which is a combination of C (category attribute label) and at least one of B (background attribute label), Q (question attribute label), O (option attribute label) and A (answer attribute label).

[0116] Among them, the data fingerprint can be a unique identification code generated by a hash algorithm (such as SHA-256) for global attribute label data, which is used to quickly compare semantic equivalence.

[0117] The comparison result may be an output value of the data fingerprint similarity analysis, which may be a matrix containing a repetition probability value (a floating point number in the range of 0-1), or a judgment result of similarity or dissimilarity.

[0118] The target data fingerprint may be the only data fingerprint retained after deduplication screening.

[0119] In some embodiments, at least one of the background attribute label, question attribute label, answer attribute label, and option attribute label of each candidate annotation data can be combined with the category attribute label to form global attribute label data, ensuring that the characteristics of each data are fully captured; then, these global attribute label data are hashed to generate a unique data fingerprint, creating an immutable identifier for each candidate annotation data; then, by comparing multiple data fingerprints, duplicate data items are identified; finally, when duplicate data fingerprints are found, deduplication processing is performed on the corresponding candidate annotation data, retaining unique data samples, thereby obtaining a non-duplicate target annotation dataset. In this way, not only the quality and uniqueness of the dataset are improved, but also the evaluation bias caused by data redundancy is avoided, thereby enabling the evaluation dataset to support more scientific and fair large language model performance evaluation.

[0120] For example, the category attribute label can be combined with the background attribute label, question attribute label, answer attribute label and option attribute label to obtain global attribute label data. Then, a hash algorithm (such as SHA3-256) is used to calculate the global attribute label data to generate a data fingerprint. For example, if there is the following candidate annotation data 1: "C": "Mathematics / Algebra", "B": "Known function f(x) = x 2 +3x+2...", "Q": "Find the minimum value of f(x) when x>0", "A": "6", "O": ["A.5", "B.6", "C.7"], the corresponding data fingerprint is calculated to be 9f86d08... (this is just an example and not calculated in detail), referred to as data fingerprint A. At this time, the preset data fingerprint set is retrieved. If the preset data fingerprint set contains a data fingerprint identical to data fingerprint A, the candidate labeled data 1 corresponding to data fingerprint A is discarded.

[0121] In some embodiments, a data fingerprint set can be used to store the data fingerprint of the global attribute label data of each candidate annotated data set corresponding to multiple candidate data sets. If a data fingerprint does not exist in the data fingerprint set, the data fingerprint is stored in the data fingerprint set; otherwise, the candidate annotated data corresponding to the data fingerprint is discarded. In other words, the data fingerprint set can be used to store the first occurrence of a data fingerprint. If a target data fingerprint that is the same as the first occurrence subsequently appears, the candidate annotated data corresponding to the subsequent target data fingerprint is directly discarded.

[0122] In some embodiments, considering that the candidate annotation data between different candidate data sets may have slight differences in expression, the threshold comparison result can be obtained by calculating the fingerprint similarity between any two data fingerprints and comparing them with a preset threshold. For example, if the fingerprint similarity between data fingerprint 1 and data fingerprint 2 is 88%, and the preset threshold is 80%, then the comparison result indicates that data fingerprint 1 and data fingerprint 2 are similar, and the candidate annotation data corresponding to one of the data fingerprints is retained, and the candidate annotation data corresponding to the other data fingerprint is deduplicated. Similarly, if there are multiple similar data fingerprints, only the candidate annotation data corresponding to one data fingerprint needs to be retained.

[0123] In some embodiments, the category attribute label and the background attribute label can be combined to obtain global attribute label data, or the category attribute label and the option attribute label can be combined to obtain global attribute label data, or the category attribute label and the option attribute label can be combined to obtain global attribute label data, and so on. The present application does not impose any specific restrictions on this. It is only necessary to keep the global candidate annotation data consistent in the way of combining to obtain global attribute label data. For example, the global attribute label data are all obtained by combining the category attribute label and the option attribute label of each candidate label data.

[0124] Through the above methods, redundant data can be eliminated quickly and effectively, and the quality of the evaluation data set can be improved, thereby avoiding evaluation bias caused by data redundancy when users use the evaluation data set for evaluation, improving user satisfaction and strengthening user stickiness.

[0125] Step 104: For each target annotated data, at least one question statement data is constructed based on the question attribute information corresponding to the question attribute tag and the option attribute information corresponding to each option attribute tag, and the subject element of each target annotated data is updated based on the at least one question statement data to obtain updated annotated data.

[0126] In some embodiments, in order to solve the problem of single evaluation dimension caused by the rigidification of data form in traditional evaluation methods, and the problem that the evaluation dataset is difficult to apply to large language models, which leads to inaccurate evaluation of the constructed evaluation dataset on the large language model, the relationship between question attribute information (Q) and option attribute information (O) can be decoupled to construct multimodal evaluation input, thereby improving the quality of the evaluation dataset.

[0127] The question attribute information may be a semantic question trunk parsed from the question attribute label (Q).

[0128] The option attribute information may be the materialized data of the option attribute tag (O).

[0129] The question statement data may be an evaluation input unit generated by combining question attribute information and option attribute information, and may be a subjective question evaluation statement formed by combining question attribute information with each option attribute information.

[0130] Among them, the updated annotation data can be the annotation data formed by replacing the original question attribute information and option attribute information in the target annotation data with the question statement data; or it can be the annotation data formed by adding the question statement data to the suffix of the original question attribute information and option attribute information in the target annotation data.

[0131] In some implementations, since objective questions generally have clear standard answers, they can ensure that the evaluation data is more objective and that different models are fairly compared under a unified benchmark. Therefore, general mainstream evaluation sets rely on objective questions to achieve efficient evaluation. However, large language models are good at generating texts that are open and semantically coherent, while the evaluation criteria for subjective questions (such as essay questions and creative writing questions) focus on logic, creativity, and semantic integrity, which are highly consistent with the text generation capabilities of large language models. Therefore, each target annotation data of this application is adapted so that the annotation data has the objectivity of objective questions and at the same time converts the annotation data into a method that large language models are good at, so as to improve the quality of the evaluation data set.

[0132] For example, for each target annotated data, the question attribute information and option attribute information can be combined and input into the large language model, requiring the large language model to directly generate the correct option. This question-answering mode is an option question-answering mode, for example, directly asking the model "2+3=? A.5, B.6"; each option can also be spliced ​​into the question in turn, converted into a subjective question, and the optimal answer is selected through model scoring. This question-answering mode is a statement scoring mode, for example, the above question is converted into "How likely is 2+3=5?", "How likely is 2+3=6?", and then the option with the highest model score is selected as the answer output by the model; the question can also be converted into a judgment question. This question-answering mode is a judgment scoring mode, for example, the question is converted into "Is 2+3=5 correct?", "Is 2+3=6 correct?", and so on. At this time, the converted sentence data "How likely is 2+3=5?", "Is 2+3=5 correct?" are question sentence data.

[0133] In some implementations, the question attribute information and option attribute information in the target annotation data can be replaced by the question statement data to form updated annotation data. For example, "How likely is 2+3=5?", "How likely is 2+3=6?" and / or "Is 2+3=5 correct?", "Is 2+3=6 correct?" in the target annotation data can be replaced by "2+3=? A.5, B.6."

[0134] Alternatively, the question statement data can be added to the suffix of the original question attribute information and option attribute information in the target annotation data. For example, "How likely is 2+3=5?", "How likely is 2+3=6?" and / or "Is 2+3=5 correct?", "Is 2+3=6 correct?" can be added to the suffix of "2+3=? A.5, B.6" in the target annotation data to form updated annotation data, etc., to increase the diversity of evaluation methods.

[0135] Through the above method, the problem of rigid evaluation format can be effectively solved, so that the generated evaluation dataset can effectively stimulate the real performance of the large language model and improve the quality of the evaluation dataset.

[0136] In some implementations, to address the issue of traditional evaluation methods having a single evaluation dimension due to the rigidity of data formats and the inability of evaluation data to fully stimulate the capabilities of large language models, the logical relationship between question attributes and option attributes can be restructured to generate multimodal question statement data, thereby enabling the evaluation dataset to meet the diverse needs of large model capability assessment. For example, step 104 of "for each target annotated data, construct at least one question statement data based on the question attribute information corresponding to the included question attribute label and the option attribute information corresponding to each option attribute label" may include:

[0137] (104.1) For each target annotated data item, determine a corresponding question-and-answer mode based on the question attribute information corresponding to the included question attribute label, the option attribute information corresponding to each option attribute label, and the answer attribute information corresponding to the answer attribute label, wherein the question-and-answer mode includes at least one of an option question-and-answer mode, a statement scoring mode, and a judgment scoring mode;

[0138] (104.2) Based on the question-answering model, the question attribute information is combined with the corresponding option attribute information to generate corresponding initial question statement data;

[0139] (104.3) Concatenate the initial question statement data with the corresponding answer attribute information to obtain at least one piece of question statement data.

[0140] Among them, the option question-answering mode can be a set of strategies that define the input form of the evaluation question, such as splicing the question attribute information and all option attribute information into a single input (such as "2+3=?A.5, B.6"), requiring the model to directly output the predicted answer, and then compare it with the preset question attribute information, thereby achieving accurate evaluation of the large language model.

[0141] Among them, the statement scoring mode can be to independently splice each option attribute information to the question attribute information and then generate multiple inputs (such as "The possibility of 2+3=5?", "The possibility of 2+3=6?", etc.), and select the predicted answer based on the confidence score of each question statement data of the large language model, and then compare it with the preset question attribute information, thereby achieving accurate evaluation of the large language model.

[0142] Among them, the judgment scoring mode can be a true or false judgment statement obtained by converting the question attribute information with the attribute information of each option (such as "Is 2+3=5 correct?", "Is 2+3=6 correct?"), requiring the large language model to output a Boolean value (T / F), and compare the predicted answer with the Boolean value "T" with the preset question attribute information, thereby achieving accurate evaluation of the large language model.

[0143] The initial question statement data may be an intermediate data unit of an unbound answer generated according to a selected mode.

[0144] For example, for the target labeled data S_i, first determine the problem attribute information Q_i corresponding to the pre-labeled problem attribute label (such as "find the function f(x) = x 2 The minimum value of ""), the option attribute information O_i corresponding to the option attribute label (such as the option set {O_i 1 = "A.0", O_i2 = "B.1", O_i3 = "C.2"}) and the answer attribute information A_i corresponding to the answer attribute label (such as "A"), so as to facilitate the selection of the object to be updated by the question.

[0145] For example, when the question-answering mode is the option question-answering mode, Q_i and O_i can be concatenated into a single input statement "Q_i+O_i" (e.g., "Find the function f(x) = x 2 The minimum value? A.0B.1C.2”) to generate the initial question statement data. Then, the initial question statement data is concatenated and bound with the answer attribute information A_i to form the final question statement data: {"question":"Find the function f(x)=x 2 A.0B.1C.2","answer":"A"}, when the large language model is evaluated based on the updated annotated data corresponding to the question sentence data, it needs to directly output the predicted option answer (such as "A").

[0146] For example, when the question-answering mode is a statement scoring mode, for the same target labeled data S_i, each option attribute information O_ij can be independently spliced ​​to the question attribute information Q_i to generate multiple initial question statement data:

[0147] {"Find the function f(x) = x 2 What is the probability that the minimum value of is 0? "}; {"Find the function f(x) = x 2 What is the probability that the minimum value of is 1? "}; {"Find the function f(x) = x 2 How likely is it that the minimum value of is 2? "}.

[0148] Alternatively, it can be in the following form:

[0149] {"output function f(x) = x 2 The minimum value is the perplexity of 0"}; {"output function f(x) = x 2 The minimum value of perplexity is 1"}; {"output function f(x) = x 2 The perplexity of"}.

[0150] Afterwards, the answer attribute information A_i ("A") can be bound to the initial question statement data. The model needs to output a confidence score or a confusion score for each statement, and finally select the option with the highest confidence score or the lowest confusion score as the predicted answer.

[0151] For example, when the question-answering model is in judgment scoring mode, for the target labeled data S_i (such as judgment questions), Q_i (such as "function f(x) = x 2 The minimum value is 0") and are spliced ​​with "T" and "F" respectively to generate the initial question statement data:

[0152] {"question":"Function f(x) = x 2 The minimum value of is 0, is this correct? ","judgment":"T"};{"question":"Function f(x)=x 2 The minimum value of is 0. Is this correct? ","judgment":"F"}.

[0153] Alternatively, it can be in the following form:

[0154] Concatenate Q_i and O_i into "Q_i+O_i" (e.g., "function f(x) = x 2 The minimum value of is 0? T / F”), the model directly outputs “T” or “F”.

[0155] Alternatively, it can be in the following form: generate two independent inputs "Q_i+T" and "Q_i+F", the model scores the two statements separately, and finally selects the option with higher confidence.

[0156] Afterwards, the answer attribute information A_i ("A") can be bound to the initial question statement data to obtain two question statement data.

[0157] Through the above approach, not only the richness and applicability of the evaluation dataset are improved, but also the flexibility and accuracy of the evaluation are enhanced, ensuring that the performance of large language models in various evaluation scenarios can be comprehensively and meticulously measured, and achieving standardization and automation of the evaluation process.

[0158] Step 105 : construct an evaluation data set based on the multiple updated annotation data, and send the evaluation data set to the target client.

[0159] In some implementations, in order to provide a solid foundation for users to perform model evaluation, an evaluation dataset can be constructed based on multiple updated annotation data to obtain a high-quality, non-redundant and structured evaluation dataset.

[0160] The evaluation data set may be a structured data set organized from a plurality of updated annotated data.

[0161] For example, multiple updated labeled data can be numbered and constructed in the order of the numbers to obtain an evaluation data set, or multiple updated labeled data can be divided according to category attribute information to construct a total evaluation data set, and then the constructed evaluation data set can be sent to the target client.

[0162] The present application receives a data set acquisition request sent by a target client, and reads a candidate data set according to the data set acquisition request, wherein the candidate data set includes multiple original data; each original data is annotated according to a preset attribute annotation rule to obtain each candidate annotated data after the annotation processing, each candidate annotated data is associated with multiple attribute labels, and the multiple attribute labels include at least a question attribute label and an option attribute label; based on the multiple attribute labels associated with each candidate annotated data, multiple candidate annotated data are deduplicated to obtain multiple target annotated data; for each target annotated data, at least one question statement data is constructed according to the question attribute information corresponding to the included question attribute label and the option attribute information corresponding to each option attribute label, and the subject element of each target annotated data is updated based on the at least one question statement data to obtain updated annotated data; an evaluation data set is constructed according to the multiple updated annotated data, and the evaluation data set is sent to the target client. In this way, the structured management of the evaluation data can be achieved by performing fine-grained attribute division and labeling on the data. This structured processing method can accurately identify and eliminate duplicate data on the core classification dimension between different data sets, and solves the redundancy problem caused by directly combining multiple data sets while retaining the data diversity characteristics. On the other hand, this application can dynamically construct diversified question statements based on attribute information and update the question elements, so that the evaluation dataset not only retains the core evaluation elements, but also generates variant data adapted to different evaluation scenarios, thereby making the structure and content of each data more consistent with the evaluation requirements of the large language model, effectively improving the coverage breadth and test validity of the evaluation dataset for the large language model capability dimension. In summary, this application can effectively remove redundant data in the dataset while improving the applicability of the reconstructed evaluation dataset.

[0163] In some embodiments, in order to enable the evaluation data set to support differentiated evaluation requirements, the evaluation level of each updated annotation data can be marked by quantifying the breadth of knowledge point coverage (knowledge dimension density) and the intensity of option interference (similarity analysis) to achieve dynamic grading and adaptive optimization of the evaluation data. For example, step 105 may include:

[0164] (105.1) Determine the corresponding knowledge dimension density index based on the number of knowledge points contained in each updated annotation data;

[0165] (105.2) Calculating the option similarity between any two option sub-data according to the multiple option sub-data included in the option attribute information corresponding to each updated annotation data, and obtaining the corresponding interference index according to the sum of the multiple option similarities corresponding to the multiple option sub-data;

[0166] (105.3) Determine the evaluation level of each corresponding updated labeled data according to the knowledge dimension density index and the interference index, and label each updated labeled data based on the evaluation level to obtain target labeled data;

[0167] (105.4) Construct an evaluation dataset based on multiple target labeled data.

[0168] The number of knowledge points may be the number of knowledge system nodes covered by a single piece of data.

[0169] Among them, the knowledge dimension density index can be a normalized indicator to characterize the breadth of data knowledge coverage.

[0170] Among them, the option sub-data can be an atomic option unit in the option attribute information (O).

[0171] The option similarity may be a measure of semantic equivalence between any two option sub-data, for example, it may be calculated using the cosine similarity of sentence vectors.

[0172] Among them, the interference index can be a comprehensive indicator of the confusingness of the quantitative option set.

[0173] The evaluation level may be a data quality grading label.

[0174] Among them, the target label data can be an enhanced data unit with additional quality evaluation metadata, and the quality label field can be expanded based on the sextuple (i.e., category attribute label, background attribute label, question attribute label, option attribute label, and answer attribute label).

[0175] In some implementations, the number of knowledge points can be normalized using a logarithmic function (e.g., a math problem covers three knowledge points, such as "quadratic function" and "extreme value calculation"), and the knowledge dimension density index K can be calculated in combination with the domain level depth.density , the specific formula is as follows:

[0176]

[0177] Among them, K density It represents the knowledge dimension density index, which is a normalized indicator used to characterize the breadth of data knowledge coverage; N represents the number of knowledge points, that is, the number of standardized knowledge graph nodes associated with a single updated annotation data (such as math problems covering knowledge points such as "quadratic function" and "extreme value calculation"); D represents the depth of the domain level, which represents the layer technology of the domain described by the knowledge point in the knowledge tree (for example, "mathematics → algebra → quadratic function" corresponds to D = 3).

[0178] Furthermore, the BERT model can be used to calculate the semantic similarity of the option sub-data and generate the interference index by weighted summation:

[0179] I disturb =∑Sim(O i ,O j ) / |Rank i -Rank j |);

[0180] Among them, O i ,O j Represents any two option sub-data, represents the option semantic unit after entity disambiguation processing (such as the option text "B.6" is mapped to the mathematical entity "NaturalNumber_6"), represents the option similarity, and is calculated by the cosine similarity of the sentence vector generated by the BERT model (range 0 to 1). For example, the similarity between "B. First pass through path a and then pass through path b" and "C. First pass through path a and then pass through path c" is 0.72 (only example), etc.; Sim(O i ,O j ) indicates the similarity of options, Rank i and Rank j Respectively represent O i and O j The option position number indicates the original arrangement order of the options in the updated annotation data (such as option "A" corresponds to Rank = 1).

[0181] Furthermore, the knowledge dimension density index and interference index can be input into a hierarchical decision tree (e.g., when the knowledge dimension density index is ≥0.7 and the interference index is ≥4.2, it is marked as level L5), and the data set can be intelligently stratified through quality labels to obtain the evaluation level of each updated annotation data. Alternatively, technical personnel can directly set the evaluation level of the corresponding updated annotation data based on the knowledge dimension density index and interference index.

[0182] Furthermore, each updated labeled data can be labeled according to the evaluation level to obtain target labeled data. For example, the target labeled data can be "C": "Mathematics / Algebra", "Q": "Find x 2 +2x+1", "O": ["A.0", "B.1", "C.-1", "D.2"], "Evaluation level": "3" (the background attribute information and answer attribute information are omitted here and are not described. In actual application, they can be increased or decreased according to specific circumstances).

[0183] Furthermore, after obtaining a plurality of target labeled data, an evaluation data set can be constructed based on the plurality of target labeled data.

[0184] Through the above method, the scientificity and effectiveness of the constructed evaluation data set can be effectively improved, which makes it easier for subsequent technical personnel to select evaluation data of different difficulty levels (i.e. evaluation levels) according to the evaluation performance of the model. For example, if the large language model performs well or excellently on the target labeled data with an evaluation level of 1, the target labeled data of level 1 can no longer be used to evaluate the large language model. The target labeled data of level 2 can be used to evaluate the large language model, and so on, thereby improving the efficiency of the evaluation.

[0185] In some embodiments, in order to enable the evaluation dataset to provide the model with clear task execution examples, without requiring the operator to compile or search for examples for each piece of data when evaluating the large language model, the dataset can be divided according to a preset ratio and a mapping relationship can be established between each piece of evaluation data and the corresponding target example data. This solves the problem of low evaluation efficiency caused by missing examples in the evaluation of large language models and helps to establish a high-quality evaluation dataset. For example, before step 105, specifically before "constructing the evaluation dataset based on multiple updated annotation data", it can also include:

[0186] (A.1) Dividing the plurality of updated annotated data into an evaluation subset and an example subset according to a preset ratio, and matching each evaluation data in the evaluation subset with the example subset based on the category attribute information and background attribute information of the evaluation data to obtain at least one target example data;

[0187] (A.2) establishing a mapping relationship between each piece of evaluation data and at least one corresponding piece of target example data;

[0188] The evaluation dataset is constructed based on multiple updated annotated data, including:

[0189] Construct an evaluation dataset based on multiple evaluation data.

[0190] The preset ratio may be a quantitative control parameter for data partitioning, such as evaluation subset: sample subset = 8:2.

[0191] The evaluation subset may be a data partition used to directly evaluate model performance.

[0192] The example subset may be a data partition used to provide contextual examples.

[0193] The evaluation data may be a single data unit in the evaluation subset.

[0194] The target example data may be example data that is retrieved from an example subset through a semantic matching algorithm and is compatible with the evaluation data.

[0195] The mapping relationship may be an association rule between the evaluation data and the target example data.

[0196] In some implementations, if the preset ratio is evaluation subset: example subset = 7:3, then, assuming the total number of all updated annotation data is 500, the evaluation subset is 350 and the example subset is 150.

[0197] In some embodiments, for each piece of evaluation data, the category attribute information and background attribute information may be concatenated, and the target example data containing category attribute labels and / or background attribute labels and keywords (e.g., kinetic energy theorem) similar to the evaluation data may be searched within the instance subset. Alternatively, the data obtained by concatenating the category attribute information and background attribute information may be matched within the example subset, and N examples with the highest similarity may be selected as the target example data.

[0198] In some implementations, an index mapping table can be established within the example subset. This table stores the mapping between each piece of evaluation data and its corresponding target example data, as well as the position of each target example data within the example subset. This facilitates rapid retrieval of the target example data corresponding to each piece of evaluation data, improving the efficiency of large language model evaluation. For example, for evaluation data D1, if its matching target example data in the example subset are E1 and E2, the index mapping table may store D1:E1,E2.

[0199] It should be noted that D1 is the position of the evaluation data in the evaluation subset. After the evaluation dataset is subsequently established, the position of the evaluation data in the evaluation dataset can remain the same, or the index mapping table can be updated based on the newly modified position. For example, if the evaluation data a is at position D1 in the evaluation subset, the mapping relationship stored in the index mapping table is D1:E1,E2; after the evaluation dataset is constructed, if the evaluation data a is still at D1 in the evaluation dataset, then the mapping relationship stored in the index mapping table is also D1:E1,E2; if the evaluation data a is at D3 in the evaluation dataset after the evaluation dataset is constructed, then the mapping relationship stored in the index mapping table is also updated accordingly, to D3:E1,E2.

[0200] In some implementations, after constructing an evaluation data set based on a plurality of evaluation data, the evaluation data set and the sample subset may be sent together to a target client.

[0201] The above approach can significantly optimize the data management and evaluation process. This dataset construction method not only simplifies the search process during evaluation, but also greatly improves the speed and accuracy of large language model evaluation.

[0202] In some embodiments, to ensure that each evaluation data can find the most relevant example data, the target example data corresponding to each evaluation data can be accurately determined by calculating the matching similarity between the data. For example, (A.1) may include:

[0203] (A.1.1) Splicing the category attribute information and background attribute information corresponding to the evaluation data to obtain a query entry;

[0204] (A.1.2) matching the query item with multiple pieces of example data in the example subset to obtain corresponding multiple matching similarities;

[0205] (A.1.3) Based on the magnitude relationship of the multiple matching similarities, determine at least one target example data corresponding to the evaluation data from the multiple example data included in the example subset.

[0206] The query item may be a semantic retrieval vector generated by combining the category attribute information and background attribute information of the evaluation data, or the query item may be a semantic retrieval vector generated by combining the category attribute label and background attribute information or background attribute label of the evaluation data.

[0207] The example data may be a single data unit in the example subset.

[0208] The matching similarity may be a quantified value of the semantic association between the query item and the sample data, and may be calculated by cosine similarity, Jaccard similarity, or the like.

[0209] For example, if the category attribute label of the evaluation data A is "mathematics / algebra" and the background attribute label is "quadratic function extreme value", then after splicing, the query item can be obtained as mathematics#algebra#quadratic function extreme value. According to the query item, matching is performed in the example subset, and multiple matching similarities with multiple example data can be matched. For example, if the matching similarity of the query item with example data E1 is 0.85, the matching similarity with example data E2 is 0.12, and the matching similarity with example data E3 is 1, if the number of target example data to be selected is 2, then example data E1 and example data E3 can be used as the target example data of evaluation data A.

[0210] In some implementations, query items may be obtained by concatenating category attribute information with background attribute information, or by concatenating category attribute information with background attribute tags, or by concatenating category attribute tags with background attribute information, or by concatenating category attribute information with background attribute tags, etc. Without departing from the spirit of the present application, the specific method for obtaining query items may be determined based on actual circumstances, and the present application does not impose any specific limitations on this.

[0211] Through the above methods, the matching accuracy and correlation between the evaluation data and the example data can be effectively improved, ensuring that each evaluation data in the evaluation data set can obtain the most appropriate context support, thereby improving the quality of the evaluation data set.

[0212] See also Figure 3 The embodiment of the present application further provides an evaluation data set construction device, which can implement the above-mentioned evaluation data set construction method. The evaluation data set construction device includes:

[0213] The receiving module 31 is configured to receive a dataset acquisition request sent by a target client and read a candidate dataset according to the dataset acquisition request, where the candidate dataset includes multiple original data.

[0214] The labeling module 32 is used to label each original data according to the preset attribute labeling rules to obtain each candidate labeled data after the labeling process, each candidate labeled data is associated with multiple attribute labels, and the multiple attribute labels include at least a question attribute label and an option attribute label;

[0215] A deduplication module 33 is configured to perform deduplication processing on multiple candidate labeled data based on multiple attribute labels associated with each candidate labeled data to obtain multiple target labeled data;

[0216] A construction module 34 is configured to construct, for each target annotated data, at least one question statement data based on the question attribute information corresponding to the question attribute tag and the option attribute information corresponding to each option attribute tag, and to update the topic element of each target annotated data based on the at least one question statement data to obtain updated annotated data;

[0217] The sending module 35 is configured to construct an evaluation data set according to the multiple updated annotation data, and send the evaluation data set to a target client.

[0218] The specific implementation of the evaluation dataset construction device is basically the same as the specific embodiment of the evaluation dataset construction method described above, and will not be repeated here. Under the premise of meeting the requirements of the embodiment of this application, the evaluation dataset construction device can also be equipped with other functional modules to implement the evaluation dataset construction method in the above embodiment.

[0219] The present application also provides a computer device comprising a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-mentioned evaluation data set construction method. The computer device can be any intelligent terminal including a tablet computer, an in-vehicle computer, or the like.

[0220] See also Figure 4 , Figure 4 The hardware structure of a computer device according to another embodiment is shown. The computer device includes:

[0221] The processor 41 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0222] The memory 42 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 42 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 42 and is called by the processor 41 to execute the evaluation data set construction method of the embodiments of this application;

[0223] Input / output interface 43, used for information input and output;

[0224] Communication interface 44, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0225] bus 45 , which transmits information between the various components of the device (e.g., processor 41 , memory 42 , input / output interface 43 , and communication interface 44 );

[0226] The processor 41 , the memory 42 , the input / output interface 43 and the communication interface 44 are connected to each other in communication within the device via a bus 45 .

[0227] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned evaluation data set construction method is implemented.

[0228] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0229] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0230] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0231] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0232] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0233] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0234] It should be understood that in this application, "at least one (item)" and "several" refer to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0235] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the above units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0236] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0237] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0238] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0239] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for constructing an evaluation dataset, characterized in that: The method comprises: Receive a dataset acquisition request sent by a target client, and read a candidate dataset according to the dataset acquisition request, wherein the candidate dataset includes a plurality of original data; Annotate each original data according to a preset attribute annotation rule to obtain each candidate annotated data after the annotation process, each candidate annotated data being associated with a plurality of attribute labels, wherein the plurality of attribute labels at least include a question attribute label and an option attribute label; Based on the multiple attribute labels associated with each candidate labeled data, performing deduplication processing on the multiple candidate labeled data to obtain multiple target labeled data; For each target annotated data, construct at least one question statement data according to the question attribute information corresponding to the question attribute tag and the option attribute information corresponding to each option attribute tag, and perform a topic element update on each target annotated data based on the at least one question statement data to obtain updated annotated data; An evaluation data set is constructed according to the plurality of updated annotation data, and the evaluation data set is sent to the target client.

2. The method for constructing an evaluation data set according to claim 1, wherein: The multiple attribute labels further include a category attribute label, an answer attribute label, and a background attribute label. Based on the multiple attribute labels associated with each candidate annotated data, deduplication processing is performed on the multiple candidate annotated data to obtain multiple target annotated data, including: Based on at least one of the background attribute label, the question attribute label, the answer attribute label, and the option attribute label of each candidate annotation data, combined with the category attribute label, global attribute label data is obtained; Performing hash calculation on the global attribute tag data to obtain the corresponding data fingerprint; Compare multiple data fingerprints to obtain comparison results; When the comparison result indicates that there are at least two target data fingerprints with the same fingerprint among the multiple data fingerprints, duplicate removal is performed on the candidate labeled data corresponding to the at least two target data fingerprints to obtain multiple target labeled data.

3. The method for constructing an evaluation data set according to claim 1, wherein: The multiple attribute tags also include an answer attribute tag. For each target annotation data, constructing at least one question statement data according to the question attribute information corresponding to the question attribute tag and the option attribute information corresponding to each option attribute tag includes: For each target annotated data, determining a corresponding question-and-answer mode based on the question attribute information corresponding to the question attribute label, the option attribute information corresponding to each option attribute label, and the answer attribute information corresponding to the answer attribute label, wherein the question-and-answer mode includes at least one of an option question-and-answer mode, a statement scoring mode, and a judgment scoring mode; Based on the question-answering mode, combining the question attribute information with the corresponding option attribute information to generate corresponding initial question statement data; The initial question statement data is concatenated with the corresponding answer attribute information to obtain at least one piece of question statement data.

4. The method for constructing an evaluation data set according to claim 1, wherein: Before constructing the evaluation data set according to the plurality of updated annotation data, the method further includes: Dividing the plurality of updated annotated data into an evaluation subset and an example subset according to a preset ratio, and matching each piece of evaluation data in the evaluation subset with the example subset according to the category attribute information corresponding to the category attribute label of the evaluation data and the background attribute information corresponding to the background attribute label, to obtain at least one target example data; Establishing a mapping relationship between each piece of evaluation data and the corresponding at least one piece of target example data; Then, constructing the evaluation data set based on the multiple updated annotation data includes: An evaluation data set is constructed according to the plurality of evaluation data.

5. The method for constructing an evaluation data set according to claim 4, wherein: The matching of the category attribute information corresponding to the category attribute label of the evaluation data and the background attribute information corresponding to the background attribute label of the evaluation data in the example subset to obtain at least one target example data includes: splicing the category attribute information and the background attribute information corresponding to the evaluation data to obtain a query entry; Matching the query item with multiple pieces of example data in the example subset to obtain corresponding multiple matching similarities; Based on the magnitude relationship of the multiple matching similarities, at least one target example data corresponding to the evaluation data is determined from the multiple example data included in the example subset.

6. The method for constructing an evaluation data set according to claim 1, wherein: The labeling process is performed on each original data according to the preset attribute labeling rules to obtain each candidate labeling data after labeling process, including: Perform attribute recognition on each raw data according to the preset attribute labeling rules to obtain the recognition result; Each original data is labeled according to the recognition result to obtain each candidate labeled data after the labeling process.

7. The method for constructing an evaluation data set according to claim 1, wherein: The step of constructing an evaluation data set based on the plurality of updated annotation data includes: Determining a corresponding knowledge dimension density index based on the number of knowledge points contained in each of the updated annotation data; Calculating the option similarity between any two option sub-data according to the plurality of option sub-data included in the option attribute information corresponding to each updated annotation data, and obtaining a corresponding interference index according to the sum of the plurality of option similarities corresponding to the plurality of option sub-data; Determining an evaluation level of each corresponding updated labeled data according to the knowledge dimension density index and the interference index, and marking each updated labeled data based on the evaluation level to obtain target labeled data; Construct an evaluation dataset based on multiple target labeled data.

8. A device for constructing an evaluation data set, characterized in that: The device comprises: A receiving module, configured to receive a dataset acquisition request sent by a target client, and read a candidate dataset according to the dataset acquisition request, wherein the candidate dataset includes a plurality of original data; A labeling module is used to label each raw data according to a preset attribute labeling rule to obtain each candidate labeled data after labeling, each candidate labeled data being associated with a plurality of attribute labels, wherein the plurality of attribute labels include at least a question attribute label and an option attribute label; a deduplication module, configured to perform deduplication processing on the plurality of candidate labeled data based on the plurality of attribute labels associated with each candidate labeled data, to obtain a plurality of target labeled data; a construction module configured to construct, for each target annotated data, at least one question statement data according to the question attribute information corresponding to the question attribute tag and the option attribute information corresponding to each option attribute tag, and perform a topic element update on each target annotated data based on the at least one question statement data to obtain updated annotated data; The sending module is used to construct an evaluation data set according to the multiple updated annotation data, and send the evaluation data set to the target client.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the evaluation data set construction method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the evaluation data set construction method according to any one of claims 1 to 7 is implemented.

Images