Method and device for expanding performance evaluation data of large language model

Abstract

The embodiment of the specification provides a large language model performance evaluation data expansion method and device, which automatically expands data on the basis of original evaluation data. For the evaluation data obtained from the evaluation data set, on the one hand, the scoring points or the deduction points are taken as seed data to generate expansion questions containing non-standardized information or even containing wrong cognition, and on the other hand, the scoring points or the deduction points are corrected according to the non-standardized information or the wrong cognition contained in the expansion questions, so as to form new evaluation data. This implementation can precisely implant logical traps in the evaluation questions, automatically construct high-quality evaluation data, and has strong expansion capability.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of computer technology, and in particular to methods and apparatus for expanding large language model performance evaluation data. Background Technology

[0002] Artificial Intelligence (AI) is a crucial component of the field of intelligence. It attempts to understand the essence of intelligence and produce new intelligent machines capable of reacting in a manner similar to human intelligence. AI is a broad science encompassing robotics, speech recognition, image recognition, natural language processing, expert systems, machine learning, and computer vision. The emergence of Large Language Models (LLMs) represents a significant leap forward in AI development. A Large Language Model (LLM), also known simply as a large model, is a natural language processing model based on deep learning techniques. Its parameters typically range from billions to hundreds of billions or even higher, possessing powerful language understanding and generation capabilities. LLMs have developed rapidly, demonstrating powerful capabilities in numerous scenarios such as natural language understanding, content creation, and intelligent question answering, and have become one of the core technologies in the field of AI. The use of LLMs typically includes two modes: direct inference and fine-tuning. In direct inference mode, users design prompts to guide the LLM to generate specific outputs. The prompts can be textual descriptions or instructions used to stimulate the semantic understanding and generation capabilities of the large language model. Here, the designed prompts can be denoted as questions, and the output of the large language model can be considered as answers.

[0003] To objectively and accurately measure the performance of different large language models and guide targeted optimization iterations, systematic evaluation of these models is typically employed. Currently, the industry generally uses standardized benchmark sets to assess the capabilities of large language models. These benchmark sets usually consist of a series of "questions" (i.e., instructions or questions input to the model) and corresponding "evaluation criteria" (i.e., pre-defined scoring standards, typically including points for scoring and points for deduction). The evaluation system scores the model's responses to the questions against these evaluation criteria, thereby quantifying the model's performance.

[0004] However, the test questions are usually designed to be simple and straightforward, making it difficult to effectively assess the ability of large language models to handle complex or non-ideal situations. Summary of the Invention

[0005] This specification describes one or more embodiments of a method and apparatus for expanding large language model performance evaluation data, in order to solve one or more problems mentioned in the background art.

[0006] According to a first aspect, a method for expanding performance evaluation data of a large language model is provided. The method includes: obtaining first evaluation data from an evaluation dataset, the first evaluation data including a first question and a first evaluation set, the first evaluation set including a plurality of score points and a plurality of deduction points; selecting first seed data from the first evaluation data according to a predetermined selection rule, wherein the first seed data includes a single score point or a single deduction point; integrating the first seed data into the first question using a large language model to obtain a first expanded question; correcting the first evaluation set based on the first expanded question; and determining the first expanded evaluation data based on the correction result and the first expanded question.

[0007] In one embodiment, the predetermined rule includes obtaining seed data in at least one of the following ways: alternately obtaining scoring points or deduction points according to a predetermined quantity or a predetermined quantity ratio; or randomly obtaining scoring points or deduction points.

[0008] In one embodiment, when the first seed data includes a single score point, the step of incorporating the first seed data into the first question using a large language model to obtain a first extended question includes: performing semantic inversion on the single score point to obtain first inversion information that contradicts the semantics of the single score point; constructing prompt information for the large language model using the first question and the first inversion information; and incorporating the first inversion information into the first question using the large language model to obtain the first extended question.

[0009] In a further embodiment, the step of modifying the first evaluation set based on the first extended question includes: adding a first deduction point or a first score point to the first evaluation set based on the first inversion information, for evaluating whether the large language model can provide indication information that the first inversion information is an incorrect premise in the question and answer of the first extended question.

[0010] In one embodiment, when the first seed data includes a single deduction point, the step of incorporating the first seed data into the first question using a large language model to obtain a first extended question includes: constructing prompt information of the large language model using the first question and the single deduction point, and incorporating the single deduction point into the first question using the large language model to obtain the first extended question.

[0011] In a further embodiment, modifying the first evaluation set based on the first extended question includes: adding a second deduction point or a second scoring point to the first evaluation set to evaluate whether the large language model can provide an indication that the single deduction point is an incorrect premise in the question and answer for the first extended question.

[0012] According to a second aspect, a method for expanding performance evaluation data for a large language model is provided. The method includes: obtaining first evaluation data from an evaluation dataset, the first evaluation data including a first question and a first evaluation set, the first evaluation set including a first set of scoring points and a first set of deduction points; selecting a target path from two candidate paths according to a predetermined selection rule, the candidate paths including a first path corresponding to the first set of scoring points and a second path corresponding to the first set of deduction points; determining first seed data based on the target path, wherein the first seed data includes scoring points or deduction points from the target path; generating a first erroneous premise based on the first seed data; incorporating the first erroneous premise into the first question to obtain a first extended question; adding a first deduction point to the first set of deduction points to correct the first evaluation set, the first deduction point being used to evaluate whether the tested large language model can indicate that the first erroneous premise is incorrect in the question-and-answer session for the first extended question; and determining first extended evaluation data to be added to the extended evaluation dataset based on the corrected first evaluation set and the first extended question.

[0013] In one embodiment, generating the first error premise based on the first seed data includes: when the first seed data is a scoring point, reversing the semantics of the first seed data to generate information that contradicts the semantics of the first seed data as the first error premise; and when the first seed data is a deduction point, generating information that is semantically consistent with the first seed information as the first error premise.

[0014] In one embodiment, the predetermined rule is one of the following: random selection; alternating selection according to a predetermined quantity; alternating selection according to a predetermined ratio.

[0015] In one embodiment, the first deduction point is the failure to clearly indicate that the first error premise is an error message.

[0016] In one embodiment, the step of generating a first erroneous premise based on the first seed data and the step of integrating the first erroneous premise into the first question to obtain a first extended question are implemented by an auxiliary large language model, including: constructing a first prompt information corresponding to a first task that generates an erroneous viewpoint based on the first seed data; constructing a second prompt information corresponding to a second task that generates an extended question based on the first question and the first task; and providing the first prompt information and the second prompt information to the auxiliary large language model, which then generates the first extended question.

[0017] According to a third aspect, an extension device for performance evaluation data of a large language model is provided. The device includes: an acquisition unit configured to acquire first evaluation data from an evaluation dataset, the first evaluation data including a first question and a first evaluation set, the first evaluation set including a plurality of score points and a plurality of deduction points; a selection unit configured to select first seed data from the first evaluation data according to a predetermined selection rule, wherein the first seed data includes a single score point or a single deduction point; a generation unit configured to integrate the first seed data into the first question using a large language model to obtain a first extended question; and a correction unit configured to correct the first evaluation set based on the first extended question.

[0018] The determination unit is configured to determine the first extended evaluation data based on the correction result and the first extended problem.

[0019] In one embodiment, the generation unit is further configured to: when the first seed data includes a single score point, semantically invert the single score point to obtain first inversion information that contradicts the semantics of the single score point; construct prompt information for a large language model using the first question and the first inversion information, and have the large language model integrate the first inversion information into the first question to obtain the first extended question; when the first seed data includes a single deduction point, construct prompt information for a large language model using the first question and the single deduction point, and have the large language model integrate the single deduction point into the first question to obtain the first extended question.

[0020] According to the fourth aspect, an extension device for large language model performance evaluation data is provided, the device comprising:

[0021] The acquisition unit is configured to acquire first evaluation data from the evaluation dataset, wherein the first evaluation data includes a first question and a first evaluation set, and the first evaluation set includes a first set of scoring points and a first set of deduction points;

[0022] The selection unit is configured to select a target path from two candidate paths according to a predetermined selection rule, and determine first seed data based on the target path. The candidate paths include a first path corresponding to a first set of scoring points and a second path corresponding to a first set of deduction points. The first seed data includes scoring points or deduction points from the target path.

[0023] The generation unit is configured to generate a first erroneous premise based on the first seed data, and to incorporate the first erroneous premise into the first problem to obtain a first extended problem;

[0024] The correction unit is configured to add a first deduction point to the first deduction point set to correct the first evaluation set. The first deduction point is used to evaluate whether the tested large language model can provide an indication that the first error premise is an error message in the question and answer of the first extended question.

[0025] The unit is configured to determine the first extended evaluation data to be added to the extended evaluation dataset based on the corrected first evaluation set and the first extended problem.

[0026] According to a fifth aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of the first or second aspect.

[0027] According to a sixth aspect, a computing device is provided, including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the method of the first aspect or the second aspect.

[0028] The methods and apparatus provided in the embodiments of this specification can be used to expand existing evaluation data to generate more realistic, non-idealized questions that may contain various flaws. Specifically, for evaluation data obtained from the evaluation dataset, on the one hand, scoring points or deduction points are used as seed data to generate expanded questions containing non-standardized information or even misconceptions; on the other hand, the corresponding scoring points or deduction points are corrected for the non-standardized information and misconceptions contained in the expanded questions, thereby forming new evaluation data.

[0029] This approach proposes a "reverse engineering" approach, starting not from the question itself, but from known evaluation criteria (i.e., scoring / deduction points). By purposefully reversing the logic of these criteria (turning correct answers into incorrect ones) or logically grafting them forward (disguising incorrect ones as correct ones), a controllable logical trap directly related to a specific knowledge point is precisely implanted into the original question. This method ensures that the newly generated questions are highly targeted, and because the process can be automatically executed based on a large language model, it can be easily applied to original evaluation sets of any domain and any size, exhibiting strong generalization capabilities. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 The screenshot shown is of an application scenario based on a large language model;

[0032] Figure 2 This is a flowchart of a method for expanding performance evaluation data of a large language model according to an embodiment of this specification;

[0033] Figure 3 A flowchart illustrating a specific example of expanding a performance evaluation dataset for a large language model is provided.

[0034] Figure 4 This is a schematic block diagram of an extension device for large language model performance evaluation data according to one embodiment of this specification. Detailed Implementation

[0035] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0036] Figure 1 This document illustrates an application scenario of the large language model. Typically, on a computing platform (computer, device, or server providing services to terminal applications, etc.) where the large language model is deployed, users can ask questions or issue commands (referred to as "questions" in this document) to the computing platform (or, in the case of a server, its client). The computing platform can then utilize the large language model's natural language processing capabilities, information augmentation retrieval, tool invocation capabilities, and answer generation capabilities to generate corresponding feedback output for the user. This feedback output can also be referred to as an answer or response.

[0037] like Figure 1In the example, a user inputs the question, "I want to lose weight, please help me create a weight loss plan." The computing platform can then use keywords from the question to provide an answer from various perspectives, including logic and health considerations. For example... Figure 1 As shown, the response from the large language model first suggests safe weight loss methods, and then provides a weight loss plan.

[0038] Understandable. Figure 1 A simple example is given, but in real-world scenarios, user questions are often more complex, potentially containing incorrect background information, hidden contradictions, ambiguous intentions, or untrue presuppositions. For instance, a user might input, "My friend recommended I lose weight quickly by not eating staple foods, and I heard it's very effective. Please design a specific diet plan for me based on this method." However, long-term abstinence from staple foods could seriously harm one's health, so this is likely a flawed viewpoint. The user's question, based on this potentially erroneous viewpoint, expects the large language model to identify this error and provide a correct and reasonable answer. Therefore, the evaluation of the large language model's response capabilities should demonstrate its ability to achieve this.

[0039] In conventional techniques, standardized benchmark sets are commonly used to evaluate the capabilities of large language models. These benchmark sets typically consist of a series of "questions" (i.e., the questions mentioned earlier) and corresponding "benchmarks" (knowledge points in the model's answers, often including points for scoring and points for deduction). The evaluation system scores the model's answers to the questions against these benchmarks, thereby quantifying the model's performance. However, with the rapid improvement in the capabilities of large language models, existing benchmark sets may have the following problems:

[0040] 1. Decreased Discrimination in Evaluation: Most existing evaluation questions are "standard questions" that are factually correct and logically clear. Large language models being evaluated, after being trained on massive amounts of data, can easily achieve high scores on these evaluation sets. With most mainstream models achieving near-perfect scores, the evaluation sets lose their function of filtering and measuring subtle differences in ability between large language models, resulting in a "ceiling effect."

[0041] 2. Detached from real-world scenarios: It cannot simulate more complex questions that users pose to the model, such as questions that may contain at least one of the following: incorrect background information, hidden contradictions, ambiguous intentions, or unrealistic presuppositions, etc.

[0042] Therefore, a high score on such a relatively simple and standardized evaluation set for a large language model does not fully represent its robustness and reliability when handling real, complex queries. For example, whether the tested large language model can identify and correct errors in user input, etc.

[0043] To address this, conventional techniques propose solutions for expanding evaluation datasets, such as: human expert construction methods and general data augmentation and perturbation methods. Human expert construction methods involve teams of domain experts, linguists, or AI researchers who, leveraging their expertise and experience, manually create challenging evaluation questions. These questions may contain complex logic, implicit conditions, cross-domain knowledge, or deliberately designed "traps," easily leading to problems such as high cost, low efficiency, strong subjectivity, and inconsistent standards. General data augmentation and perturbation methods borrow data augmentation techniques from traditional Natural Language Processing (NLP), performing synonym replacement, syntactic structure transformation (e.g., changing active sentences to passive sentences), random word insertion or deletion, and rewriting interpretations using other models on existing simple questions. However, this approach may suffer from problems such as blind perturbation, lack of specificity, and disruption of question semantics and evaluation standards, undermining the validity of the original evaluation data and potentially increasing the cost of subsequent manual verification. Furthermore, the potential for increasing the complexity of the evaluation data is often limited.

[0044] In view of this, this specification provides a scheme for expanding large language model performance evaluation data, automatically expanding the large language model performance evaluation data. Specifically, for evaluation data obtained from existing evaluation datasets, on the one hand, the scoring points or deduction points in the evaluation criteria are used as seed data to generate extended questions containing non-standardized information and even misconceptions. On the other hand, for the non-standardized information and misconceptions contained in the extended questions, the corresponding scoring points or deduction points are corrected, thereby forming new evaluation data. In this way, starting from known evaluation criteria (i.e., scoring points / deduction points), controllable logical traps directly related to specific knowledge points are generated in reverse and included in the questions, ensuring that the newly generated questions are both highly targeted and highly generalizable.

[0045] The technical concept of this specification is described in detail below with reference to the accompanying drawings.

[0046] Figure 2 This specification illustrates an extended process for evaluating the performance of a large language model using an embodiment of the data. The process can be executed by a computer, device, or server with sufficient computing power. This process can be based on an existing evaluation dataset. The existing evaluation dataset is used to evaluate the performance of a large language model in responding to user questions (including user questions, task instructions, etc.).

[0047] like Figure 2 As shown, the expansion process of the large language model performance evaluation data provided in this specification may include: Step 201, obtaining first evaluation data from the evaluation dataset, the first evaluation data including a first question and a first evaluation set, the first evaluation set including several scoring points and several deduction points; Step 202, selecting first seed data from the first evaluation data according to a predetermined selection rule, wherein the first seed data includes a single scoring point or a single deduction point; Step 203, using the large language model to integrate the first seed data into the first question to obtain a first extended question; Step 204, correcting the first evaluation set based on the first extended question; Step 205, determining the first extended evaluation data based on the correction result and the first extended question.

[0048] First, in step 201, the first evaluation data is obtained from the evaluation dataset.

[0049] As can be understood, an evaluation dataset can be a dataset used to test a large language model in answering user questions or executing user instructions. An evaluation dataset can include multiple evaluation data points. Each evaluation data point can include a user question and evaluation criteria, each of which is represented here by an evaluation set. User questions are the instructions or questions input by the user, while evaluation criteria are the key points in the large language model's answer. Key points that should be answered are recorded as scoring points, and key points that should not appear but appear are recorded as deduction points. Therefore, an evaluation set can include one or more scoring points and one or more deduction points.

[0050] The evaluation dataset here can be any existing evaluation dataset for large language models. It can be a domain-specific evaluation dataset (such as infant care, fitness and wellness, etc.) or a general dataset; there are no restrictions here.

[0051] The first evaluation data can be any piece of evaluation data in the evaluation dataset. The user question corresponding to the first evaluation data is denoted as the first question, and the corresponding evaluation set is denoted as the first evaluation set. In an optional embodiment, the first evaluation set can also be divided into a score point set and a deduction point set, where the score point set can be denoted as the first score point set, and the deduction point set can be denoted as the first deduction point set.

[0052] As an example, the first set of evaluation data includes, for instance: the first question, "Please describe how to safely bathe an infant"; the first set of scoring points {maintain room temperature between 24-26℃ to avoid the infant getting cold; choose a time when the infant is awake and not hungry, such as 30 minutes after feeding; ensure water temperature is between 37-40℃; someone must supervise the infant throughout the bath and never leave them unattended; support the head and neck; choose tear-free formula baby bath wash and shampoo with a suitable pH level; ...}; and the first set of deductions {using adult bath wash; recommending not dressing the infant after bathing; etc.}. The first set of scoring points and the first set of deductions together constitute the first evaluation set.

[0053] Next, in step 202, the first seed data is selected from the first evaluation data according to the predetermined selection rules.

[0054] Under the technical concept of this specification, reverse engineering is employed. Starting from known evaluation points (i.e., scoring points and / or deduction points in the evaluation set), controllable logical traps directly related to specific knowledge points are generated and incorporated into the problem. To this end, the knowledge points to be modified, i.e., scoring points and / or deduction points, must be selected from the first set of evaluation data. For a single set of evaluation data, multiple sets of extended data can be generated. This specification describes the selection of one evaluation point as an example. The selected evaluation point is denoted as the first seed data.

[0055] It is understood that the first seed data comes from the scoring points or deduction points in the first evaluation set. In order to select the first seed data, in one embodiment, scoring points or deduction points can be randomly selected; in another embodiment, scoring points or deduction points can be selected alternately in a predetermined number (such as 1 or 2); in yet another embodiment, scoring points or deduction points can be selected alternately in a predetermined ratio (such as 2:1), and so on, without limitation.

[0056] The first seed data point could be a scoring point or a deduction point. For example, regarding the first question, "Please describe how to safely bathe a baby," a specific example of the sampled first seed data point could be the scoring point, "Ensure the water temperature is between 37-40℃."

[0057] Then, in step 203, the first seed data is incorporated into the first problem using a large language model to obtain the first extended problem.

[0058] To reverse-engineer user questions and test the large language model, selected scoring or deduction points can be incorporated into the user questions as incorrect premises or misleading logic. This allows us to first determine the negative semantics corresponding to the first seed data. Here, negative semantics can be unreasonable or incorrect semantic information such as description errors, contradictions of common sense, logical errors, or not being the correct answer in the user question, rather than negative semantics.

[0059] The negative semantic information can be generated manually or by a large language model. This approach leverages the overall language processing capabilities of the large language model, avoiding the harshness of negative semantic information, and also avoids consuming additional human resources.

[0060] It's understandable that a scoring point is inherently a positive semantic answer to a question. To obtain corresponding negative semantics, semantic inversion can be performed to obtain inverted information that contradicts the semantics of the scoring point. The inverted information obtained for the first seed data is denoted as the first inverted information. Here, semantic contradiction means mutual conflict or containing erroneous information, and it doesn't necessarily need to cover all situations outside the scoring point, nor does it necessarily need to completely discard the scoring point. For example, for the scoring point "ensure the water temperature is between 37-40℃," the corresponding first inverted information could be "the water temperature is greater than 45℃," or "the water temperature is 46℃," or "the water temperature is any temperature," all of which can be understood as corresponding negative semantic information.

[0061] For a deduction point, it is data containing error information, such as using adult shower gel to bathe an infant. Therefore, in some embodiments, the deduction point itself can be negative semantic information. In optional embodiments, when the first seed data includes deduction points, in order to obtain richer negative semantics, the deduction point can be processed through a large language model to generate corresponding negative semantic information. For example, the negative semantic information generated for the deduction point "It is recommended not to wear clothes after bathing" is "Do not dress the infant within two hours after bathing," etc. The large language model used here can be the large language model to be evaluated, or other large language models; this specification does not limit this.

[0062] Furthermore, the negative semantics corresponding to the first seed data can be incorporated into the first question to generate an extended question, denoted as the first extended question. This process can be performed using a large language model. Specifically, the first question and the generation task can be used as prompts for the large language model, which then performs semantic fusion using its own language processing capabilities to obtain the first extended question.

[0063] As a concrete example, the prompts from a large language model could be:

[0064] The original question was: "Please describe how to safely bathe a baby."

[0065] Negative semantic information: "Ensure the water temperature is greater than 45℃";

[0066] Task Description: Please naturally incorporate the above negative semantic information into the original question, making the new question appear as if a real user is asking it with this flawed premise.

[0067] The feedback result from the large language model might be, for example, "I've heard that it's better to bathe a baby in warmer water, such as above 45 degrees Celsius, to kill germs. Given this, what other safety precautions should be taken when bathing an infant?" This feedback result can serve as the first extension question. It's worth noting that the large language model here can still be any large language model.

[0068] In some possible designs, the process can be executed in two steps using a large language model to generate the first extension question. The first step generates negative semantic information, and the second step generates the extension question. Below is a specific example of the prompt information input to a large language model:

[0069] "Step 1: Generate a false opinion."

[0070] - The correct knowledge point is: "{The water temperature should be kept between 37-40 degrees Celsius}".

[0071] Based on the correct knowledge points mentioned above, please generate a contradictory viewpoint that sounds like a common misconception.

[0072] Step 2: Incorporate the erroneous viewpoint into the original question.

[0073] The original question was: "Please describe how to safely bathe a baby."

[0074] Please naturally incorporate the flawed viewpoint generated in the first step into the original question, making the new question appear as if a real user is asking the question with this flawed premise.

[0075] Final requirement: Please only output the rewritten new problem.

[0076] In this way, regardless of whether the first seed data is a scoring point or a deduction point, its corresponding negative semantic information can be integrated into the first question to generate the first extended question. Among them, generating extended questions through a large language model is more rounded and natural, and more in line with the possible input of real users, compared with solutions such as splicing according to a pre-defined template, keyword replacement based on pre-defined rules, and direct injection of simple paraphrasing.

[0077] Next, in step 204, the first evaluation set is modified based on the first extended problem.

[0078] It's understandable that after modifying the first question to form the first extended question, the addition of negative semantic information makes it more likely that the large language model can better identify errors or inconsistencies in the responses. Therefore, the first evaluation set needs to be modified accordingly, adding relevant scoring or deduction points to assess the large language model's ability to recognize errors in user questions.

[0079] For the first evaluation set, a scoring point can be added, designated as the first scoring point, or a deduction point can be added, designated as the first deduction point. The first scoring point or the first deduction point is used to evaluate whether the large language model can indicate in the question-and-answer session for the first extended question that the incorporated negative semantic information (such as the inverse information corresponding to a single scoring point or a single deduction point) is an incorrect premise. For example, indicating that the negative semantic information is an incorrect premise. Accordingly, a scoring point can be added stating "Explicitly points out...is incorrect," or a deduction point can be added stating "Does not explicitly point out...is incorrect," and so on.

[0080] As an example, in the previous example, if the first extended question were "I heard that it's better to bathe a baby in warmer water, such as above 45 degrees Celsius, to kill germs. Under this premise, what other safety precautions should be taken when bathing an infant?", then the scoring point could be added for "clearly stating that bathing a baby in water above 45 degrees Celsius is wrong," and the deduction point could be added for "failing to clearly state that bathing a baby in water above 45 degrees Celsius is wrong," and so on.

[0081] The correction of the first evaluation set can be done manually or automatically generated according to predetermined rules. For example, "It is wrong not to explicitly state that..." (... indicates negative semantic information in the preceding text) can be added to the first deduction point set. It can also be corrected through a large language model, which will not be elaborated here.

[0082] Then, in step 205, the first extended evaluation data is determined based on the correction results and the first extended problem.

[0083] The revised first evaluation set and the first extended question can be added to the first evaluation dataset as the first extended evaluation data. In practice, further targeted extensions can be made based on the first evaluation data at other scoring or deduction points, which will not be elaborated here.

[0084] In possible designs, expansion can be based on scoring points and deduction points, denoted as different expansion paths. The expansion path can then be selected according to predetermined rules to determine whether to select scoring points or deduction points. The predetermined rules can be switching between the two paths according to a predetermined number or proportion, or they can be randomly selected. To clarify the technical concept of this specification, Figure 3This document illustrates a schematic diagram of an overall implementation architecture for expanding large language model evaluation data, as provided in this specification.

[0085] like Figure 3 As shown, when expanding the evaluation data of a large language model, we can first obtain a single evaluation data point from the evaluation dataset. This evaluation data point can include the original question and the evaluation set. The original question, denoted as the first question, can be any single evaluation data point. Correspondingly, the evaluation set can be denoted as the first evaluation set, which includes a first set of scoring points and a first set of deduction points. The first set of scoring points corresponds to several scoring points, and the first set of deduction points corresponds to several deduction points. Then, according to predetermined rules, path selection can be performed, choosing either path 1 or path 2 as the target path. Path 1 and path 2, as candidate paths, can also be denoted as the first path and the second path, respectively. Path 1 can correspond to the first set of scoring points, and path 2 can correspond to the first set of deduction points. The specific selection of path 1 or 2 can be based on predetermined rules. Predetermined rules can be, for example, one of the following: random selection; alternating selection according to a predetermined number; alternating selection according to a predetermined ratio; etc.

[0086] By following the target path, the first seed data can be selected and the first erroneous premise can be generated. Incorporating this first erroneous premise into the first question yields the first extended question. Furthermore, a corresponding deduction point is added to the first deduction point set, denoted as the first deduction point, to correct the first evaluation set. The first deduction point is used to evaluate whether the tested large language model can indicate that the first erroneous premise is incorrect in its response to the first extended question. For example, the first deduction point might be: "Failed to clearly indicate that the first erroneous premise is incorrect."

[0087] It is understandable that, when the first seed data is a scoring point, the semantics of the first seed data can be reversed to generate information (opinions) that contradict the semantics of the first seed data as the first incorrect premise. When the first seed data is a deduction point, information that is semantically consistent with the first seed information can be generated as the first incorrect premise.

[0088] In a possible implementation, generating the incorrect premise and the first extended question can be performed via an auxiliary large language model. In this case, a first prompt message corresponding to the first task of generating the incorrect viewpoint can be constructed based on the first seed data; then, a second prompt message corresponding to the second task of generating the extended question can be constructed based on the first question and the first task; the first and second prompt messages are then provided to the auxiliary large language model, which generates the first extended question. This auxiliary large model can be the large language model being evaluated, or it can be another large language model.

[0089] refer to Figure 3As shown, when path 1 is selected, expansion can be performed based on the score points in the first score point set. Specifically, a target score point A can be sampled from the first score point set as the first seed data, assuming A represents a correct fact. Rewriting instructions are constructed based on the original question and score point A. These instructions can include first and second prompts. The first prompt can be used to complete the first task of generating an incorrect viewpoint A' based on the false premise that A's content is incorrect. The second prompt can include the second task of generating a first extended question based on the fusion of the incorrect viewpoint A' and the first question. These rewriting instructions can be understood as instructions to invoke the auxiliary large language model, i.e., prompts provided to the auxiliary large model.

[0090] Then, the auxiliary large model is invoked to execute rewriting instructions, generating a new question that takes the erroneous viewpoint A' as an erroneous premise—the first extended question. The new question can contain the erroneous viewpoint A', which contradicts A. Furthermore, a first deduction point can be added to check whether the answer generated by the large language model indicates that A' is an erroneous premise. If it fails to identify A' as an erroneous premise, a deduction can be made from the evaluated large language model.

[0091] Still referencing Figure 3 As shown, when path 2 is selected, expansion can be made based on the deduction points in the first deduction point set. Specifically, a target deduction point C is sampled from the first deduction point set as the first seed information. Here, it is assumed that C represents an erroneous fact. A rewriting instruction is constructed based on the original question and the deduction point C, wherein the rewriting instruction includes at least a second prompt. Based on the assumption that the content contained in C is correct, the second prompt can correspond to the second task of integrating the erroneous viewpoint into the first question. That is, a rewriting instruction that calls the large language model is constructed based on the erroneous viewpoint C. In an optional embodiment, C can also be modified to generate an erroneous viewpoint C' consistent with the viewpoint of C. This can be a specific example of C, such as C being "the bath water temperature is greater than 45℃", and C' including "the bath water temperature is 60℃". In this case, the rewriting instruction can also include a first prompt, which corresponds to the first task of generating the erroneous viewpoint C' consistent with C. Then, the auxiliary large model is called to execute the rewriting instruction to generate a new question containing information consistent with information C.

[0092] Furthermore, deduction points can be added to determine whether the answer generated by the evaluated large language model can point out that the information corresponding to C or C' is incorrect. Points can be deducted from the evaluated large language model if it fails to point out the corresponding error.

[0093] Regardless of whether path 1 or path 2 is executed, new questions and new evaluation criteria (including scoring points and deduction points) can be combined to form a new set of evaluation data, which can be denoted as the first extended evaluation data. Figure 3 As shown, the data is output to a high-quality evaluation dataset. This evaluation data is a targeted modification of the original evaluation data, and the extended questions are automatically optimized using a large language model. Based on the optimization results, the evaluation criteria are also corrected by adding deduction points. It has higher complexity and higher quality, and is more in line with the various situations that real users may encounter when using a computing platform with a large language model.

[0094] Reviewing the above process, we can expand the existing evaluation data to generate more realistic, non-idealized questions that may contain various flaws. Specifically, for the evaluation data obtained from the evaluation dataset, on the one hand, we use the scoring points or deduction points as seed data to generate expanded questions containing non-standardized information and even misconceptions. On the other hand, we correct the corresponding scoring points or deduction points for the non-standardized information and misconceptions contained in the expanded questions, thereby forming new evaluation data. This solution, through reverse engineering, starts from the evaluation points based on existing simple, standard evaluation data, performing purposeful logical reversal (turning correct into incorrect) or logical forward grafting (disguising incorrect as correct), precisely embedding logical traps in the evaluation questions, automatically constructing high-quality evaluation data, and possessing strong scalability.

[0095] More specifically, the technical solutions provided in this specification can solve one or more of the following technical problems and achieve the corresponding technical effects:

[0096] To address issues such as high costs and low efficiency, a fully automated process is adopted. Raw evaluation datasets can be processed in batches by calling large language model APIs. This provides an automated and scalable technical path that can transform massive amounts of basic evaluation data into high-value complex evaluation data at a lower cost and higher efficiency, accelerating the iteration and enrichment of evaluation resources and reducing the cost and cycle of building high-quality evaluation sets.

[0097] To address the issues of strong subjectivity and inconsistent standards in expert-generated questions, standardized instructions and pre-set modification paths ensure the systematicness and reproducibility of the question modification process, guaranteeing consistency in style, difficulty gradient, and logical trap types of generated questions, which is superior to manual methods that rely on personal experience.

[0098] To address issues such as limited coverage, a systematic approach can be taken to traverse every scoring and deduction point in the original evaluation set and modify it to achieve comprehensive and fine-grained coverage of knowledge points. This approach can uncover complex scenario combinations that are difficult for human experts to conceive of. By generating a large number of complex questions containing hidden and real logical traps, it can effectively distinguish the subtle differences between different models in advanced intelligence (such as critical thinking and fact-checking ability) and solve the problems of "high score inflation" and "ceiling effect" caused by simple evaluation sets.

[0099] To address issues such as indiscriminate and untargeted perturbations, evaluation data is constructed in reverse from the evaluation criteria. This goes beyond simply modifying surface-level text without a clear objective. Instead, it involves purposefully inserting erroneous information that contradicts specific correct knowledge points. This transforms each new question into a precise "probe" to test the model's ability to identify and correct errors related to that knowledge point. This expands the evaluation focus from traditional "knowledge retrieval and answering" capabilities to higher-level knowledge identification and correction abilities. Furthermore, each complex question has specific evaluation criteria, providing clear guidance for optimizing the large language model and effectively expanding the evaluation dimensions for comprehensively assessing the overall intelligence level of the large language model.

[0100] To address issues such as the potential for disrupting question semantics and evaluation criteria, this approach guides the large language model through "intelligent reconstruction" rather than "brute-force modification." The generated questions effectively simulate complex queries that users might pose in the real world, containing erroneous information or inappropriate premises. This enhances the real-world simulation capability of the evaluation, ensuring that the modified questions retain the original intent while adding a controllable number of logical traps. By simultaneously enhancing evaluation criteria, it guarantees that new questions always have self-consistent and effective evaluation standards, avoiding the problem of standard failure. This allows for a more accurate measurement of the robustness, reliability, and security of the large language model in practical applications; and so on.

[0101] According to another embodiment, an expansion device for large language model performance evaluation data is also provided, which can be located in a computer, device, or server with certain computing power.

[0102] Figure 4 An expansion device 400 for large language model performance evaluation data is shown according to one embodiment. For example... Figure 4 As shown, the expansion device 400 for large language model performance evaluation data may include: an acquisition unit 401, a selection unit 402, a generation unit 403, a correction unit 404, and a determination unit 405. This device 400 can be installed on any computer, device, or server with a certain computing capability.

[0103] According to a possible design: an acquisition unit 401 is configured to acquire first evaluation data from an evaluation dataset, the first evaluation data including a first question and a first evaluation set, the first evaluation set including several scoring points and several deduction points; a selection unit 402 is configured to select first seed data from the first evaluation data according to a predetermined selection rule, wherein the first seed data includes a single scoring point or a single deduction point; a generation unit 403 is configured to integrate the first seed data into the first question using a large language model to obtain a first extended question; a correction unit 404 is configured to correct the first evaluation set based on the first extended question; and a determination unit 405 is configured to determine the first extended evaluation data based on the correction result and the first extended question.

[0104] In one possible embodiment, the first seed data includes a single score point, and the generation unit 403 can be further configured to: perform semantic inversion on the single score point to obtain first inversion information that contradicts the semantics of the single score point; construct prompt information of a large language model using the first question and the first inversion information; and have the large language model integrate the first inversion information into the first question to obtain a first extended question.

[0105] In another possible embodiment, the first seed data includes a single deduction point, and the generation unit 403 can be further configured to: construct prompt information of a large language model using the first question and the single deduction point, and have the large language model integrate the single deduction point into the first question to obtain a first extended question.

[0106] According to another possible design: A selection unit 401 is configured to acquire first evaluation data from an evaluation dataset, the first evaluation data including a first question and a first evaluation set, the first evaluation set including a first set of scoring points and a first set of deduction points; a selection unit 402 is configured to select a target path from two candidate paths according to a predetermined selection rule, and determine first seed data based on the target path, the candidate paths including a first path corresponding to the first set of scoring points and a second path corresponding to the first set of deduction points, the first seed data including scoring points or deduction points from the target path; a generation unit 403 is configured to generate a first incorrect premise based on the first seed data, and to integrate the first incorrect premise into the first question to obtain a first extended question; a correction unit 404 is configured to add a first deduction point to the first set of deduction points to correct the first evaluation set, the first deduction point being used to evaluate whether the tested large language model can provide an indication that the first incorrect premise is incorrect in the question-and-answer session for the first extended question; a determination unit 405 is configured to determine, based on the corrected first evaluation set and the first extended question, to add the first extended evaluation data to the extended evaluation dataset.

[0107] It is worth noting that, Figure 4 The device 400 shown is Figure 2 or Figure 3The method embodiment shown corresponds to this, therefore, Figure 2 or Figure 3 The corresponding descriptions in the method embodiments can also be applied to... Figure 4 The device 400 shown will not be described in detail here.

[0108] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 2 , Figure 3 The methods described above.

[0109] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 2 , Figure 3 The methods described above.

[0110] Those skilled in the art will recognize that the functions described in the embodiments of this specification in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0111] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the technical concept in this specification. It should be understood that the above description is only a specific embodiment of the technical concept in this specification and is not intended to limit the scope of protection of the technical concept in this specification. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments in this specification should be included within the scope of protection of the technical concept in this specification.

Claims

1. An extension method of large language model performance evaluation data, the method comprising: obtaining first evaluation data from an evaluation data set, the first evaluation data comprising a first question and a first evaluation set, the first evaluation set comprising a plurality of scoring points and a plurality of deduction points; selecting first seed data from the first evaluation data according to a predetermined selection rule, wherein the first seed data comprises a single scoring point or a single deduction point; integrating the first seed data into the first question in the form of an error premise or misleading logic by using a large language model to obtain a first extended question; correcting the first evaluation set based on the first extended question; determining first extended evaluation data according to the correction result and the first extended question.

2. The method of claim 1, wherein, The predetermined selection rule comprises obtaining seed data in at least one of the following ways: alternately obtaining scoring points or deduction points according to a predetermined number or a predetermined number ratio; randomly obtaining scoring points or deduction points.

3. The method of claim 1, wherein, In the case where the first seed data comprises a single scoring point, the step of integrating the first seed data into the first question in the form of an error premise or misleading logic by using a large language model to obtain a first extended question comprises: performing semantic inversion on the single scoring point to obtain first inverted information that is contrary to the semantics of the single scoring point; constructing prompt information of a large language model by using the first question and the first inverted information, and integrating the first inverted information into the first question by the large language model to obtain the first extended question.

4. The method of claim 3, wherein, The step of correcting the first evaluation set based on the first extended question comprises: based on the first inverted information, adding a first deduction point or a first scoring point in the first evaluation set for evaluating whether the large language model can give an indication that the first inverted information is an error premise in an answer to the first extended question.

5. The method of claim 1, wherein, In the case where the first seed data comprises a single deduction point, the step of integrating the first seed data into the first question in the form of an error premise or misleading logic by using a large language model to obtain a first extended question comprises: constructing prompt information of a large language model by using the first question and the single deduction point, and integrating the single deduction point into the first question by the large language model to obtain the first extended question.

6. The method of claim 5, wherein, The step of correcting the first evaluation set based on the first extended question comprises: adding a second deduction point or a second scoring point in the first evaluation set for evaluating whether the large language model can give an indication that the single deduction point is an error premise in an answer to the first extended question.

7. An extension method of large language model performance evaluation data, the method comprising: obtaining first evaluation data from an evaluation data set, the first evaluation data comprising a first question and a first evaluation set, the first evaluation set comprising a first scoring point set and a first deduction point set; selecting a target path from two candidate paths according to a predetermined selection rule, the candidate paths comprising a first path corresponding to the first scoring point set and a second path corresponding to the first deduction point set; determine first seed data based on the target path, wherein the first seed data comprises a scoring point or a deduction point derived from the target path; generate a first error premise based on the first seed data; integrate the first error premise into the first question to obtain a first extended question; add a first deduction point to the first deduction point set to correct the first evaluation set, and the first deduction point is used to evaluate whether the tested large language model can give an indication that the first error premise is incorrect information in the question and answer for the first extended question; determine first extended evaluation data based on the corrected first evaluation set and the first extended question and add the first extended evaluation data to the extended evaluation data set.

8. The method of claim 7, wherein, The generating a first error premise based on the first seed data comprises: in the case that the first seed data is a scoring point, inversely semantically processing the first seed data to generate information inconsistent with the semantics of the first seed data as the first error premise; in the case that the first seed data is a deduction point, generating information consistent with the semantics of the first seed information as the first error premise.

9. The method of claim 7, wherein, The predetermined selection rule is one of the following: random selection; alternating selection according to a predetermined number; and alternating selection according to a predetermined proportion.

10. The method of claim 7, wherein, The first deduction point is a failure to clearly indicate that the first error premise is incorrect information.

11. The method of claim 7, wherein, The generating a first error premise based on the first seed data and the integrating the first error premise into the first question to obtain a first extended question are implemented by an auxiliary large language model, comprising: constructing first prompt information corresponding to a first task of generating an error viewpoint based on the first seed data; constructing second prompt information corresponding to a second task of generating an extended question based on the first question and the first task; providing the first prompt information and the second prompt information to an auxiliary large language model, and generating the first extended question by the auxiliary large model.

12. An apparatus for extending large language model performance evaluation data, the apparatus comprising: an acquisition unit configured to acquire first evaluation data from an evaluation data set, the first evaluation data comprising a first question and a first evaluation set, and the first evaluation set comprising a plurality of scoring points and a plurality of deduction points; a selection unit configured to select first seed data from the first evaluation data according to a predetermined selection rule, wherein the first seed data comprises a single scoring point or a single deduction point; a generation unit configured to integrate the first seed data into the first question in the form of an error premise or misleading logic by a large language model to obtain a first extended question; a correction unit configured to correct the first evaluation set based on the first extended question; a determination unit configured to determine first extended evaluation data based on the correction result and the first extended question.

13. The apparatus of claim 12, wherein the generation unit is further configured to: In the case that the first sub-data includes a single score point, the single score point is semantically reversed to obtain first reversed information opposite to the semantics of the single score point; construct prompt information of a large language model by using the first question and the first inverted information, and integrate the first inverted information into the first question by the large language model to obtain the first extended question. In a case where the first seed data includes a single penalty point, prompt information of the large language model is constructed by using the first question and the single penalty point, the single penalty point is integrated into the first question by the large language model, and the first extended question is obtained.

14. An extension device for large language model performance evaluation data, the device comprising: an acquisition unit configured to acquire first evaluation data from an evaluation data set, the first evaluation data comprising a first question and a first evaluation set, the first evaluation set comprising a first score point set and a first penalty point set; a selection unit configured to select a target path from two candidate paths according to a predetermined selection rule, and determine first seed data based on the target path, the candidate paths comprising a first path corresponding to the first score point set and a second path corresponding to the first penalty point set, the first seed data comprising a score point or a penalty point derived from the target path; a generation unit configured to generate a first error premise according to the first seed data, and integrate the first error premise into the first question to obtain a first extended question; a correction unit configured to add a first penalty point to the first penalty point set to correct the first evaluation set, the first penalty point being used to evaluate whether the tested large language model can give an indication that the first error premise is incorrect information in the answer to the first extended question; a determination unit configured to determine first extended evaluation data according to the corrected first evaluation set and the first extended question, and add the first extended evaluation data to an extended evaluation data set.

15. A computer readable storage medium having a computer program stored thereon, the computer program, when executed in a computer, causing the computer to perform the method of any one of claims 1-11.

16. A computing device comprising a memory and a processor, wherein: The memory stores executable code, and the processor executes the executable code to implement the method of any one of claims 1-11. The memory stores executable code, and the processor executes the executable code to implement the method of any one of claims 1-11.

Images