A model evaluation method and apparatus

Abstract

The application discloses a model evaluation method and device, the method comprises the following steps: obtaining a first data entry and a second data entry in a sports health knowledge base; based on the first data entry, a test question and a corresponding standard answer are constructed; according to the first data entry, the second data entry and the third data entry, each different type of test context is constructed; through a to-be-tested large language model, based on each different type of test context and test question, each to-be-tested answer corresponding to the test question is determined; according to each to-be-tested answer and the standard answer, the to-be-tested large language model is evaluated to obtain an evaluation result. The method can find the loyalty problem of the to-be-tested large language model before it goes online, so as to reduce the safety risk of the sports health content generated by the to-be-tested large language model.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model evaluation method and apparatus. Background Technology

[0002] With the development of artificial intelligence technology, large language models are increasingly being applied to the field of sports and health. Based on user-inputted exercise goals, physical conditions, or rehabilitation needs, they can generate exercise plans, rehabilitation suggestions, exercise risk warnings, and nutritional recommendations. Because sports and health content is highly specialized, to improve the accuracy of the generated content, Retrieval-Augmented Generation (RAG) is typically employed. This involves retrieving relevant sports and health data from a sports and health knowledge base and inputting the retrieved data as contextual information into the large language model, enabling the model to generate corresponding sports and health content based on this contextual information.

[0003] In the process of generating sports and health content using a large language model, the large language model may not strictly follow the context of the input to answer questions, resulting in responses that do not meet user needs. In other words, the large language model may have a faithfulness problem.

[0004] Currently, the usual approach is to collect real user input of exercise and health questions, the exercise and health content generated by the large language model, and the context information on which that content was generated, after the large language model is deployed, to determine the fidelity of the large language model (i.e., whether the large language model has generated illusory content that does not match the context information). However, the above method is difficult to detect fidelity issues before the large language model under test is deployed, resulting in significant security risks to the exercise and health content generated by the large language model under test. Summary of the Invention

[0005] In view of this, embodiments of this application provide a model evaluation method and apparatus that can detect fidelity issues in a large language model before it is launched online, thereby reducing the security risks of sports and health content generated by the large language model under test.

[0006] In a first aspect, embodiments of this application provide a model evaluation method, the method comprising: Obtain the first data entry and the second data entry from the sports and health knowledge base, wherein the second data entry and the first data entry satisfy a preset semantic matching condition; Based on the first data entry, construct test questions and corresponding standard answers; Based on the first data entry, the second data entry, and the third data entry, test contexts of different types are constructed respectively, wherein the third data entry is obtained by modifying the first data entry; Using the large language model to be tested, based on the various test contexts and the test questions, determine the corresponding answers to the test questions; The large language model to be tested is evaluated based on the individual test answers and the standard answers to obtain the evaluation results.

[0007] Secondly, embodiments of this application provide a model evaluation apparatus, the apparatus comprising: The acquisition module is used to acquire a first data entry and a second data entry from the sports and health knowledge base, wherein the second data entry and the first data entry satisfy a preset semantic matching condition; The question construction module is used to construct test questions and corresponding standard answers based on the first data entry; The test scenario construction module is used to construct test contexts of different types based on the first data entry, the second data entry, and the third data entry, wherein the third data entry is obtained by modifying the first data entry. The answer module is used to determine the corresponding answers to the test questions based on the various test contexts and the test questions, using the large language model to be tested. The evaluation module is used to evaluate the large language model to be tested based on each answer to be tested and the standard answer, and obtain the evaluation result.

[0008] Thirdly, embodiments of this application provide a device, the device comprising: Memory, used to store computer programs; A processor for executing the computer program to cause the device to perform the model evaluation method described in the first aspect above.

[0009] Fourthly, embodiments of this application provide a computer storage medium on which a computer program is stored. When the computer program is run, a device running the computer program implements the model evaluation method described in the first aspect above.

[0010] Compared with the prior art, the embodiments of this application have the following beneficial effects: This application provides a model evaluation method, which includes: acquiring a first data entry and a second data entry from a sports and health knowledge base that meet preset semantic matching conditions, and constructing a test question and its corresponding standard answer based on the first data entry, so that the test content can match the data in the actual sports and health knowledge base; constructing different types of test contexts based on the first data entry, the second data entry, and a third data entry obtained by modifying the first data entry, so that different test scenarios can be formed around the same test question, such as the existence of answer basis, interference from similar information, and changes in data content; then determining the test answer based on each different type of test context using the large language model under test, and evaluating the large language model under test based on each test answer and the standard answer, so as to actively detect whether the large language model under test can faithfully answer based on the currently provided sports and health data in different test scenarios, thereby discovering the fidelity problem of the large language model under test before it goes online, and reducing the security risks caused by the large language model under test generating inaccurate sports and health content. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart of a model evaluation method provided in an embodiment of this application; Figure 2 A schematic diagram of the architecture of a model evaluation method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the overall processing flow of a model evaluation method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a model evaluation device provided in an embodiment of this application. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0014] Because sports and health content is highly specialized and may involve information such as exercise intensity, frequency, recovery period, and contraindications, retrieval-enhanced generation techniques can be used to improve the accuracy of the generated content. This involves retrieving sports and health data related to the user's needs from a sports and health knowledge base and inputting the retrieved data as contextual information into a large language model, which then generates corresponding sports and health content based on this contextual information.

[0015] In the process of generating sports and health content using large language models, the models may not strictly adhere to the input context information. Instead, they may infer based on knowledge learned during training, or generate content that is not present in the context or inconsistent with it. In other words, large language models may have fidelity issues. For example, even when the context information does not contain relevant sports and health knowledge, the large language model may still generate specific exercise suggestions on its own. Or, even when key information such as training time or number of training sessions changes in the context, the large language model may still generate sports and health content that has not changed accordingly based on its existing knowledge.

[0016] Currently, the typical approach involves collecting real user input of exercise and health questions, the exercise and health content generated by the large language model, and the context information used to generate that content after the model is deployed. Then, through manual review or automated detection, the consistency between the generated content and the context information is determined. However, this method relies on actual operational data generated after the large language model is deployed. The model's fidelity to the specific exercise and health scenario can only be tested after a real user triggers the corresponding scenario. Therefore, existing technologies struggle to detect fidelity issues before the large language model is deployed, leading to significant security risks in the exercise and health content generated by the model.

[0017] Based on this, to solve the above problems, in this embodiment of the application, a first data entry and a second data entry that meet the preset semantic matching conditions in the sports and health knowledge base can be obtained, and a test question and its corresponding standard answer can be constructed based on the first data entry. This can make the test content match the data in the actual sports and health knowledge base. By constructing different types of test contexts based on the first data entry, the second data entry, and a third data entry obtained by modifying the first data entry, different test scenarios can be formed around the same test question, such as the existence of answer basis, interference from similar information, and changes in data content. Then, the test answer is determined by the large language model under test based on each different type of test context, and the large language model under test is evaluated based on each test answer and the standard answer. In different test scenarios, it is possible to actively detect whether the large language model under test can faithfully answer based on the currently provided sports and health data. This can identify the fidelity problem of the large language model under test before it is launched online, and reduce the security risks caused by the large language model under test generating inaccurate sports and health content.

[0018] The specific implementation of the model evaluation method and apparatus in this application will be described in detail below with reference to the accompanying drawings and through embodiments.

[0019] See Figure 1 The figure is a flowchart of a model evaluation method provided in an embodiment of this application, combined with... Figure 1 As shown, it can specifically include: S101: Obtain the first data entry and the second data entry from the sports and health knowledge base. The second data entry and the first data entry satisfy the preset semantic matching conditions.

[0020] The sports and health knowledge base can be used by the large language model under test. It can store exercise prescriptions, rehabilitation guidelines, nutritional advice, exercise risk warnings, exercise contraindications, and other sports and health data. Data entries are data units in the sports and health knowledge base that can independently represent a piece of sports and health knowledge. They can be text data or content extracted from tables, documents, or other structured or unstructured data.

[0021] The first data entry is the target data entry used to construct the test questions and standard answers. The second data entry is a data entry that is semantically related to the first data entry and used to create interference. The third data entry is a data entry obtained by modifying at least one key piece of information in the first data entry. The first, second, and third data entries are used to distinguish data entries with different functions and are not used to limit the order in which the data entries are obtained.

[0022] In this embodiment, a data entry to be evaluated can be selected from a sports and health knowledge base as the first data entry. The first data entry can be specified by the tester or automatically selected from the sports and health knowledge base according to a preset order, data category, update time, or risk level. For example, the first data entry can be a piece of data from an exercise prescription, post-operative rehabilitation training recommendations, or sports nutrition recommendations.

[0023] The second data entry is a data entry in the sports and health knowledge base that is semantically related to the first data entry. Preset semantic matching conditions are used to determine the second data entry that is semantically similar to the first data entry. For example, the preset semantic matching conditions may include: the semantic similarity between the first and second data entries is greater than or equal to a preset similarity threshold; or, the second data entry belongs to the top K data entries obtained after sorting candidate data entries according to semantic similarity, where K is a positive integer. The second data entry can discuss the same or similar sports and health topics as the first data entry, but may differ in training actions, execution time, number of training sessions, applicable targets, or other key information.

[0024] Optionally, it can be further determined that the second data entry cannot be used alone to uniquely determine the standard answer corresponding to the test question. In this way, the second data entry can serve as real interference information from the actual sports and health knowledge base, and its language expression, professional content, and data structure match the actual retrieval results after the large language model under test is launched, thereby improving the consistency between the test scenario and the actual business scenario.

[0025] S102: Based on the first data entry, construct the test question and the corresponding standard answer.

[0026] The test question is used to ask for at least one key piece of information from the first data entry, and the standard answer is the content in the first data entry that corresponds to that key piece of information. For example, you can first identify objects, actions, times, frequency, intensity, constraints, and other entities or attributes in the first data entry, and then select at least one key piece of information that can be explicitly answered, and generate the test question and standard answer accordingly.

[0027] For example, it can be verified that the answer to a test question can be uniquely determined by a first data entry, and that the answer cannot be uniquely determined solely by a second data entry. Building on this, questions whose answers are not easily determined by the general knowledge of the language model under test can be selected, so that the language model under test needs to actually utilize data from the current test context to answer.

[0028] S103: Construct test contexts of different types based on the first data entry, the second data entry, and the third data entry. The third data entry is obtained by modifying the first data entry.

[0029] The test context is the data content input to the large language model under test along with the test question. It can consist of one or more data entries, representing the data conditions that the large language model under test can obtain when answering the test question. A third data entry is a data entry obtained by modifying at least one key piece of information from the first data entry. The third data entry has the same or similar theme as the first data entry, and at least one key piece of information in the third data entry changes relative to the first data entry. For example, different types of test contexts can be constructed by retaining, removing, or replacing the first data entry with a third data entry, allowing each type of test context to represent different data conditions.

[0030] For example, different types of test contexts can reuse the same second data entry and use the same test question. Thus, by primarily changing the data content related to the availability of the standard answer across different test contexts, additional variables introduced by differences in interfering information or test questions can be reduced. This facilitates determining whether the large language model under test faithfully responds based on the currently provided data, according to the differences between the various test answers.

[0031] When constructing the test context, the individual data entries can be organized as continuous text without adding additional tags, and the order of the data entries can be randomly adjusted. This reduces the impact of obvious prompts on the response process of the large language model under test, making the test context closer to the contextual information actually provided to the large language model in the retrieval enhancement generation process.

[0032] S104: Using the large language model to be tested, determine the corresponding answers to the test questions based on various test contexts and test questions of different types.

[0033] For example, each test context can be combined with the same test question to form multiple independent model inputs, and each of these inputs can be invoked on the large language model under test. The large language model under test generates a test answer for each model input, thus obtaining test answers corresponding one-to-one with different types of test contexts. To reduce state interference between different invocations, each invocation can use independent model sessions with the same model parameters and answer requirements. The test answer is the answer generated by the large language model under test based on a test context and a test question.

[0034] The large language model to be tested refers to a large language model that needs to undergo fidelity testing. For example, it can be a generative model in a retrieval enhancement generation system, or it can be a general large language model that only receives the provided test context and test questions. This application embodiment does not specifically limit it. Fidelity refers to the characteristic of the large language model to be tested to generate answers based on the currently provided test context, not outputting content lacking current data support, and updating the answers accordingly when the current data changes. By actively constructing test data based on the sports and health knowledge base before the large language model to be tested goes online, fine-grained evaluation can be carried out around each first data item in the sports and health knowledge base without waiting for real users to trigger the corresponding scenario.

[0035] S105: Based on each answer to be tested and the standard answer, evaluate the large language model to be tested and obtain the evaluation results.

[0036] For example, based on each answer to be tested and the standard answer, it can be determined whether each answer to be tested meets the expected answer requirements in the corresponding test context, and an evaluation result can be generated. The evaluation result may include a correct or incorrect identifier for each answer to be tested, the degree of semantic consistency with the standard answer, the reference or support for the test context, the fidelity result of the large language model to be tested for a single first data item, and the evaluation statistics for multiple first data items.

[0037] When different types of test contexts represent scenarios with sufficient data, missing data, and data content changes, the evaluation results can respectively reflect the ability of the tested large language model to extract the correct answer when information is sufficient, its ability to avoid generating unfounded answers when information is missing, and its ability to update answers based on current data when key information changes. Furthermore, the evaluation results can be correlated with the corresponding first data item to identify high-risk data items that are likely to cause the tested large language model to generate incorrect answers, providing a basis for subsequent adjustments to the sports and health knowledge base, optimization of retrieval strategies, or selection of the tested large language model.

[0038] Therefore, by acquiring the first and second data entries that meet the preset semantic matching conditions from the sports and health knowledge base, and constructing test questions and their corresponding standard answers based on the first data entries, the test content can be matched with the data in the actual sports and health knowledge base. By constructing different types of test contexts based on the first data entries, the second data entries, and the third data entries obtained by modifying the first data entries, different test scenarios can be formed around the same test question, such as the existence of answer basis, interference from similar information, and changes in data content. Then, the test answer is determined by the large language model under test based on each different type of test context, and the test answer and the standard answer are evaluated. In different test scenarios, it is possible to actively detect whether the large language model under test can faithfully answer based on the currently provided sports and health data. This allows for the discovery of fidelity issues before the large language model under test goes online, without relying on user logs, and reduces the security risks caused by the large language model under test generating inaccurate sports and health content.

[0039] In one possible implementation, test contexts of three types can be constructed: one for data integrity, one for data missing, and one for data modification. A test context of data integrity can be constructed based on a first data entry and a second data entry. This test context is used to test the response capability of the large language model under test when complete data is included.

[0040] The test context for the data integrity type may include a first data entry and at least one second data entry. The first data entry includes content that can be used to determine the standard answer corresponding to the test question; the second data entry satisfies a preset semantic matching condition with the first data entry, but the second data entry cannot be used alone to uniquely determine the standard answer.

[0041] The data integrity type test context is used to simulate a scenario where the search results include both data entries that support the correct answer and other data entries that are semantically similar to that data entry. In this case, the large language model under test needs to identify the first data entry from multiple semantically related data entries and answer the test question based on the content of the first data entry.

[0042] As an example, the first data entry is "Patients after knee surgery should begin straight leg raise training in the second week post-surgery, 10 repetitions per set, 3 sets per day," and the second data entry is "Patients after knee surgery should begin isometric quadriceps contraction training in the third week post-surgery, holding for 5 seconds each time, 4 sets per day." The first and second data entries can be used together to construct a test context of data completeness type. If the test question asks for the start time and number of repetitions of straight leg raise training, the large language model under test should output "Starting in the second week post-surgery, 10 repetitions per set, 3 sets per day" based on the first data entry.

[0043] If the test language model outputs the same answer as the standard answer, it indicates that the test language model can identify the first data entry used to determine the answer even in the presence of semantically related data interference. If the test answer does not match the standard answer, it indicates that the test language model failed to correctly identify or use the key information in the first data entry.

[0044] Furthermore, based on the second data entry, a test context for data missing types can be constructed. This test context is used to test the response capability of the large language model under test in the case of data missing.

[0045] A test context with missing data may include a second data entry, but not a first data entry. While the second data entry satisfies the preset semantic matching conditions with the first data entry, it cannot be used alone to uniquely determine the standard answer to the test question. Therefore, the test context with missing data lacks complete data to support that standard answer.

[0046] The test context for data missing types is used to simulate situations where the retrieval results fail to recall the target data entry used to determine the answer, and only recall other data entries semantically similar to the target data entry. In this case, if the large language model under test faithfully answers based on the currently provided test context, the large language model under test can indicate that it cannot determine the answer to the test question based on the current data, and should not generate a specific answer based on knowledge learned during its training process or similar content in the second data entry.

[0047] Taking the aforementioned sports rehabilitation scenario as an example, a test context with missing data could include only the second data entry related to isometric quadriceps contraction training, excluding the first data entry related to straight leg raise training. In this case, the test language model should indicate that it cannot determine the start time and number of repetitions for straight leg raise training based on the given data. If the test language model still outputs "Starting in the second week post-surgery, 10 repetitions per set, 3 sets per day" or other specific answers, it indicates that the test language model generated content lacking current data support.

[0048] Furthermore, based on the second and third data entries, a test context of data modification type can be constructed. This test context of data modification type is used to test the response ability of the large language model under test when the data is modified.

[0049] Test contexts of the data modification type can include the second and third data entries, but not the first data entry. These test contexts simulate situations where the data in the sports and health knowledge base has been updated, and the large language model under test needs to answer based on the updated data. In this case, the large language model under test should generate the answer based on the modified content in the third data entry, and should not continue to output the unmodified content corresponding to the first data entry.

[0050] As an example, we can modify "week 2 post-surgery" to "week 3 post-surgery" in the first data entry, and change "10 times per group" to "15 times per group" to obtain the third data entry. This third data entry and the second data entry can then be used together to construct a test context for the data modification type. In this case, the large language model under test should output "Starting from week 3 post-surgery, 15 times per group, 3 groups per day". If the large language model under test still outputs the original standard answer, it indicates that it is not faithfully using the currently provided third data entry.

[0051] The three different types of test contexts described above can correspond to the same test question and can reuse the same second data entry. This allows the main difference between the three test contexts to be the presence, absence, or replacement by the third data entry of the first data entry used to determine the answer, reducing the impact of other test condition differences on the evaluation results.

[0052] Therefore, by constructing test contexts for data completeness, missing data, and modified data, the ability of the large language model under test to identify the correct answer in complete data, avoid generating unfounded answers when data is missing, and update the answer based on the current data when data is modified can be tested separately. This allows for the evaluation of the fidelity of the large language model under test from multiple complementary test scenarios, thereby improving the comprehensiveness of the fidelity evaluation results.

[0053] In one possible implementation, key information in the first data entry can be modified by modifying the model to obtain a third data entry for constructing a test context of the data modification type, and the standard answer can be updated synchronously based on the third data entry. Specifically, the key information in the first data entry is modified by modifying the model to obtain the third data entry.

[0054] The modification model is used to identify key information in the first data entry that corresponds to the test question, and to modify at least one piece of key information. The modification model can be a large language model, a natural language processing model trained on a text modification task, or a combination of a rule processing module and a text generation model; this application does not specifically limit it in this regard.

[0055] Key information may include the subject of the exercise or rehabilitation, training movements, start time, duration, number of repetitions, number of sets, frequency, exercise intensity, nutritional intake, contraindications, or applicable conditions, etc., which can affect the answers to the test questions.

[0056] When modifying the first data entry, the sports and health theme, main target, basic sentence structure, and other unselected content can be kept unchanged. Only the target key information corresponding to the test question can be modified. This allows the third data entry to maintain a similar overall semantics to the first data entry, while simultaneously changing the correct answer to the test question.

[0057] As an example, the first data entry is "Patients after knee surgery should begin straight leg raise training in the second week post-surgery, 10 repetitions per set, 3 sets per day." The test question asks for the start time of straight leg raise training, the number of repetitions per set, and the number of sets per day. Modifying the model can change "second week post-surgery" to "third week post-surgery" and "10 repetitions per set" to "15 repetitions per set," resulting in the third data entry, "Patients after knee surgery should begin straight leg raise training in the third week post-surgery, 15 repetitions per set, 3 sets per day."

[0058] Furthermore, a validity check can be performed on the third data entry. For example, it can be checked whether the third data entry conforms to the preset format of sports and health data, whether the third data entry can still be used to clearly answer test questions, and whether the answer corresponding to the third data entry differs from the standard answer. When a third data entry fails the validity check, the key information to be modified can be re-identified, or the third data entry can be regenerated by modifying the model.

[0059] The standard answer can also be updated based on the third data entry to obtain an updated answer, which is then used to evaluate the large language model under test.

[0060] The updated answer includes the modified information in the third data entry corresponding to the content asked in the test question. It can identify the content in the standard answer that corresponds to the modified key information, and replace the corresponding content in the standard answer with the modified information from the third data entry to obtain the updated answer.

[0061] Alternatively, the third data entry and the test question can be input into the modification model or the question construction model, and the updated answer can be regenerated through the modification model or the question construction model. That is, regardless of the method used, the updated answer should be consistent with the current content in the third data entry.

[0062] Taking the above example, the original standard answer was "Starting from the second week after surgery, 10 times per set, 3 sets per day". After obtaining the third data entry, the standard answer can be updated to "Starting from the third week after surgery, 15 times per set, 3 sets per day", and this answer can be used as the updated answer.

[0063] The updated answer can be used to evaluate the test answer in the context of the data modification type. If the test answer is consistent with the updated answer, it indicates that the large language model under test can answer based on the current content in the third data item; if the test answer is still consistent with the standard answer before modification, it indicates that the large language model under test has failed to update the answer according to the change in data content.

[0064] Therefore, by modifying the key information in the first data entry and updating the standard answer synchronously according to the third data entry, it is possible to construct test conditions in which the correct answer has changed while keeping the theme of sports and health and the data expression method basically unchanged. This allows us to test whether the large language model under test faithfully uses the currently provided data, rather than relying on existing memory or data before modification to generate the answer.

[0065] In one possible implementation, an evaluation model can be used to synthesize the various test answers, standard answers, and updated answers, and evaluate the large language model under test according to the test context type corresponding to each test answer. Specifically, the evaluation model can be used to evaluate the large language model under test based on each test answer, standard answer, and updated answer to obtain the evaluation result.

[0066] The evaluation model can be a large language model, a natural language processing model trained on text classification or semantic matching tasks, or a combination of a rule-based decision-making module and a large language model. The evaluation model is used to determine whether each answer meets the expected response requirements in the corresponding test context.

[0067] For example, the test question, each answer to be tested, the standard answer, the updated answer, and the test context type corresponding to each answer to be tested can be input into the evaluation model, so that the evaluation model can make judgments according to the evaluation rules corresponding to the data integrity type, data missing type, and data modification type, respectively.

[0068] For data integrity-type test contexts, the evaluation model can compare the corresponding test answer with the standard answer to determine whether the test answer includes the key information represented by the standard answer. If the test answer and the standard answer are semantically consistent, the test answer can be determined to meet the response requirements for data integrity; if they are inconsistent, the test answer can be determined to not meet the response requirements for data integrity.

[0069] For test contexts with missing data, the evaluation model can determine whether the corresponding answer indicates that the answer cannot be determined based on the currently provided data, and whether the answer contains specific sports and health content that lacks support from the test context. If the answer correctly indicates that the current data is insufficient, then the answer meets the requirements for responding to a data-missing type; if the answer still outputs a specific answer despite the absence of the first data entry, then the answer does not meet the requirements for responding to a data-missing type.

[0070] For test contexts involving data modification, the evaluation model can compare the corresponding test answer with the updated answer and determine whether the test answer still includes the original content from the standard answer. If the test answer and the updated answer are semantically consistent, the test answer meets the requirements for a data modification response; if the test answer remains consistent with the standard answer but inconsistent with the updated answer, the test answer does not faithfully use the third data entry.

[0071] When making comparisons, the evaluation model allows for expressive differences between the test answer and the standard or updated answer that do not affect the actual meaning. For example, "starting in the second week after surgery" and "starting in the second week after surgery" can be determined to be semantically consistent without requiring the two answers to be exactly the same string.

[0072] The evaluation results may include a correct or incorrect label for each answer, the reasoning behind the judgment, confidence level, or score. The evaluation results may also include the correct answer rate for data completeness, the correct rejection rate for data missing, the correct answer rate for data modification, and a comprehensive fidelity index determined based on the above indicators.

[0073] As an example, if the large language model under test outputs "Starting from the second week after surgery, 10 times per group, 3 groups per day" for the test context of the data completeness type, "Cannot be determined based on the current data" for the test context of the data missing type, and "Starting from the third week after surgery, 15 times per group, 3 groups per day" for the test context of the data modification type, then the evaluation model can determine that the above three answers meet the answer requirements under the corresponding test context.

[0074] Therefore, by evaluating each answer to be tested according to the evaluation requirements corresponding to different test contexts, and combining the standard answer and the updated answer, the different natural language expressions of the answers to be tested can be uniformly processed, and different errors generated by the large language model to be tested under the conditions of data integrity, data missing and data modification can be identified. This reduces the workload of manually reviewing each answer to be tested and improves the efficiency of the evaluation of the large language model to be tested before it goes online.

[0075] In one possible implementation, for multiple different large language models to be tested, the same test context and test questions of different types can be used to determine each test answer, and the evaluation results can be obtained separately according to the same evaluation method, so as to compare the fidelity performance of multiple large language models to be tested.

[0076] Multiple different large language models to be tested can be large language models with different network structures, parameter sizes, training data, training methods, or versions, or they can be the same large language model configured with different model parameters or different prompts.

[0077] For each large language model to be tested, different types of test contexts and test questions can be input into the large language model to be tested, so that the large language model to be tested can output the corresponding test answer for each test context.

[0078] When testing multiple different large language models, the first, second, and third data entries, test questions, standard answers, and updated answers can be fixed. The same temperature parameters, maximum output length, and response format requirements can also be used. This allows multiple large language models to be tested under the same exercise and health data conditions, reducing the impact of different test data or model calling conditions on the comparison results.

[0079] As an example, test contexts for data integrity, missing data, and modified data can be input into the first and second large language models under test, respectively. The first and second large language models under test each output three test answers for the three test contexts, thus obtaining two sets of test answers.

[0080] For each large language model to be tested, corresponding evaluation results can be generated based on its individual test answers and standard answers. If the updated answer has already been determined, the test answers corresponding to the test context of the data modification type can also be evaluated in conjunction with the updated answer.

[0081] Furthermore, the evaluation results can be statistically analyzed according to the large language model under test, the test context type, the category of the first data entry, or the risk level. For example, the correct answer rate of each large language model under test can be calculated separately under the data completeness type, the correct rejection rate under the data missing type, and the correct answer rate under the data modification type. Based on the above statistical results, comparison results between multiple large language models under test can be generated.

[0082] As an example, the evaluation metrics for the first large language model under test were 95%, 80%, and 90% for data completeness, missing data, and modified data, respectively, while the corresponding evaluation metrics for the second large language model under test were 92%, 50%, and 85%. Based on the above evaluation results, it can be determined that both large language models under test have good answering ability when the information is complete, but the first large language model under test has a higher correct rejection rate when the data is missing, and therefore has better fidelity performance.

[0083] Therefore, by enabling multiple different large language models to generate test answers based on the same test question and various test contexts, and obtaining evaluation results according to consistent evaluation rules, comparable evaluation of multiple large language models under the same sports and health data conditions can be achieved, providing a basis for the selection of large language models for deployment, version replacement, or parameter configuration.

[0084] In one possible implementation, after obtaining the evaluation results, the evaluation results of each test case can be statistically analyzed based on the evaluation results, according to the data group type and / or the large language model under test, to determine the accuracy of the large language model under test under different data conditions, and generate the corresponding model evaluation report.

[0085] Each test case can correspond to one of the following types: data completeness, data missingness, or data modification. For any given statistical object, the total number of test cases included in that object and the number of correctly answered test cases can be counted. The accuracy rate is then determined based on the ratio between the number of correctly answered test cases and the total number of test cases.

[0086] Optionally, the accuracy rates for complete data types, missing data types, and modified data types can be calculated separately according to data type.

[0087] Specifically, all test cases in the data integrity type can be statistically analyzed to determine the number of first test cases in which the language model under test (LMT) answers correctly. The first accuracy rate for the data integrity type is then determined based on the ratio of the number of first test cases to the total number of test cases in the data integrity type. The first accuracy rate characterizes the LMT's ability to answer test questions correctly based on the provided data, given sufficient data. Similarly, all test cases in the data missing type can be statistically analyzed to determine the number of second test cases in which the LMT correctly refuses to answer. The second accuracy rate for the data missing type is then determined based on the ratio of the number of second test cases to the total number of test cases in the data missing type. Since the test context for the data missing type does not include data entries that can be used to determine the answer to the test question, the LMT in the data missing type can be considered correct if it indicates that it cannot determine the answer based on the current data; conversely, if the LMT outputs a specific answer lacking support from the current data, it can be considered incorrect. The second accuracy rate characterizes the LMT's ability to correctly refuse to answer when information is insufficient; its ideal value is 100%. Secondly, a higher accuracy rate indicates that the large language model under test is better able to avoid generating specific answers when there is a lack of answer basis, and thus can reflect the fidelity of the large language model under test to the provided data.

[0088] Furthermore, statistics can be compiled on all test cases within the data modification type to determine the number of third test cases in which the language model under test correctly answers based on the modified data. The third accuracy corresponding to the data modification type is then determined based on the ratio between the number of third test cases and the total number of test cases in the data modification type. Within the test context corresponding to the data modification type, if the language model under test outputs the test answer that matches the modified data and the corresponding updated answer, the answer is considered correct; if the language model under test still outputs the standard answer corresponding to the data before modification, the answer is considered incorrect. The third accuracy characterizes whether the language model under test can faithfully generate answers based on the currently provided modified data, rather than relying on memories formed during training or the data before modification. A higher third accuracy indicates that the language model under test is more sensitive to changes in the currently provided data, and its answers are more consistent with the modified data.

[0089] Optionally, statistics can be performed according to the dimensions of the large language model under test. For the same sports and health test dataset, tests can be performed using multiple different large language models under test, and the accuracy of each large language model under test can be calculated in terms of data completeness, missing data, and modified data.

[0090] When performing statistical analysis on multiple large language models under test, corresponding model evaluation information can be generated for each model. This evaluation information may include the model identifier, model version, total number of test cases in each data set, number of correctly answered test cases, and accuracy rates for data integrity, missing data, and modified data types.

[0091] Furthermore, a multi-model comparison report can be generated based on the model evaluation information corresponding to each of the multiple large language models under test. The multi-model comparison report can use tables, bar charts, line charts, radar charts, or other visualization formats to show the accuracy of different large language models under test in various datasets.

[0092] Therefore, we can intuitively compare the ability of different large language models under test to give normal answers when sufficient data is available, their ability to correctly refuse to answer when information is insufficient, and their ability to faithfully use the current data when the data is modified.

[0093] In one possible implementation, a preset search algorithm can be used to retrieve data entries from the sports and health knowledge base that meet preset semantic matching conditions with the first data entry, and the retrieved data entries can be used as the second data entries. For example, an embedding model can be used to convert each data entry in the sports and health knowledge base into a corresponding entry vector, and each data entry and its entry vector can be associated and stored in a vector database.

[0094] Here, the entry vector can be a set of numerical values ​​used to represent the semantic features of the corresponding data entry. The more similar the semantics of different data entries, the smaller the distance between their corresponding entry vectors in the vector space, or the higher the similarity between them.

[0095] The embedding model can be a neural network model trained on a text semantic representation task. When a sports and health knowledge base is established or updated, the embedding model processes each data entry and stores the resulting entry vectors in a vector database. When new or updated data entries are added to the sports and health knowledge base, the corresponding entry vectors can be generated or updated synchronously.

[0096] Then, the first entry vector corresponding to the first data entry is determined, and a candidate entry vector similar to the first entry vector is retrieved from the vector database using a preset search algorithm.

[0097] For example, the first data entry can be converted into a first entry vector using the same embedding model used to generate other entry vectors. Then, the similarity between the first entry vector and other entry vectors in the vector database can be calculated, and candidate entry vectors can be determined based on the similarity. Alternatively, the first entry vector corresponding to the first data entry can be directly queried from the vector database, and then the similarity between the first entry vector and other entry vectors in the vector database can be calculated, and candidate entry vectors can be determined based on the similarity.

[0098] The preset search algorithm can be an Approximate Nearest Neighbor (ANN) algorithm. For example, the top K entry vectors from the vector database can be selected as candidate entry vectors according to their similarity to the first entry vector, from high to low, where K is a positive integer. The preset search algorithm can also be an exact nearest neighbor (ANN) algorithm or other algorithms capable of semantic retrieval; this application does not specifically limit this.

[0099] Next, the data entry corresponding to the candidate entry vector can be determined as the second data entry. For example, the data entry corresponding to at least one candidate entry vector can be determined as the second data entry.

[0100] As an example, the first data entry relates to "straight leg raise training after knee surgery." A preset search algorithm can retrieve candidate data entries related to "isometric contraction training of the quadriceps after knee surgery" and "ankle pump training after knee surgery." Both of these candidate data entries relate to post-knee rehabilitation, but their training movements, start times, and repetitions differ. Therefore, these candidate data entries can be designated as the second data entry.

[0101] When multiple second data entries are identified, each second data entry satisfies a preset semantic matching condition with the first data entry and can be used together to construct various types of test contexts. The similarity between each second data entry and the first data entry can also be recorded to control the semantic similarity of interfering data in the test context.

[0102] Therefore, by using a preset search algorithm to retrieve data entries from the sports and health knowledge base that meet the preset semantic matching conditions with the first data entry, the second data entry can be constructed directly using real data from the actual business environment of the large language model under test. This ensures that the professional content, expression, and data distribution of the second data entry are consistent with the actual retrieval results, thereby reducing the bias introduced by manually written interference data and improving the ability of the evaluation results to reflect the fidelity risk in real business scenarios.

[0103] In one possible implementation, a test question and corresponding standard answer can be constructed based on a first data entry using a question construction model, and the standard answer can include the content of the first data entry. For example, candidate key information in the first data entry can be identified using the question construction model.

[0104] The question construction model can be a large language model or a natural language processing model trained on a question generation task or an information extraction task. The question construction model can perform semantic analysis on the first data entries and identify candidate key information from the first data entries that can be used to form test questions and standard answers.

[0105] Candidate key information may include the target of exercise or rehabilitation, training movements, start time, duration, number of repetitions, number of sets, frequency, exercise intensity, nutritional intake, contraindications, and applicable conditions. For example, for the first data entry "Patients after knee surgery should begin straight leg raise training in the second week after surgery, 10 repetitions per set, 3 sets per day", the problem construction model can identify candidate key information such as "patients after knee surgery", "second week after surgery", "straight leg raise training", "10 repetitions per set" and "3 sets per day".

[0106] Furthermore, a test question can be constructed based on at least one candidate key information using a question construction model, and the standard answer can be determined based on the first data entry.

[0107] Test questions can be used to ask for a single candidate key piece of information, or multiple related candidate key pieces of information simultaneously. Test questions can be in the form of natural language questions, and their meaning should be clear to avoid multiple different reasonable answers.

[0108] The standard answer includes the content of the first data entry and corresponds to the candidate key information requested by the test question. In other words, the standard answer should be directly identifiable from the first data entry without requiring inference from other sports and health knowledge outside of the first data entry.

[0109] Taking the first data item mentioned above as an example, the problem construction model can construct the test question "Based on the given data, in which week after surgery should the patient start straight leg raise training? How many times per set? How many sets per day?" and determine "Starting in the second week after surgery, 10 times per set, 3 sets per day" as the corresponding standard answer.

[0110] Furthermore, priority can be given to candidate key information that is difficult to determine the answer based solely on the general knowledge learned by the large language model under test during training. For example, specific training time, number of training sessions, or exercise intensity can be prioritized, so that the large language model under test needs to actually use the first data entry in the test context to accurately answer the test question.

[0111] In addition, the validity of the test questions and standard answers can be checked.

[0112] Validity checks can be used to determine whether the test questions and standard answers are suitable for evaluating the large language model under test. Specifically, this can be checked whether the standard answer can be uniquely identified by the first data entry, whether the standard answer fully covers the content asked by the test questions, and whether the test questions have clear and unambiguous expressions.

[0113] It can also be checked whether the standard answer cannot be uniquely determined based solely on the second data entry. If the standard answer can also be directly determined by the second data entry, the test context for data missing types may still contain complete answer evidence, thus affecting the judgment of the test large language model's ability to respond in the case of data missingness.

[0114] If any validity check fails, the candidate key information used to construct the test question can be adjusted, or the question construction model can reconstruct the test question and the standard answer. For example, if the test question asks for both training time and the number of training iterations, but the standard answer only includes training time, the standard answer can be supplemented or the test question can be reconstructed; if the second data entry can also determine the standard answer, the second data entry can be replaced or new candidate key information can be selected.

[0115] Therefore, by identifying candidate key information in the first data item through a question construction model, and automatically constructing test questions and standard answers based on the candidate key information, test cases can be generated in batches around different data items in the sports and health knowledge base. Furthermore, by performing validity checks on the test questions and standard answers, it can be ensured that there is a clear answer correspondence between the test questions and the first data item, and that the standard answer cannot be determined independently for the second data item. This improves the effectiveness of the test cases, expands the coverage of the pre-deployment evaluation of the sports and health knowledge base, avoids test coverage being limited by user behavior distribution, comprehensively covers key data items in the sports and health knowledge base, and supports targeted verification capabilities for specific data, enabling fine-grained fidelity evaluation of each data item.

[0116] In one possible implementation, refer to Figure 2 , Figure 2 This is a schematic diagram illustrating the architecture of a model evaluation method provided in an embodiment of this application. Figure 2 As shown, the system may include a sports and health knowledge base server 101, a vector database server 102, a large language model server to be tested 103, an evaluation large language model server 104, and a client terminal 105. The various servers and the client terminal 105 can establish communication connections via wired or wireless networks.

[0117] The sports and health knowledge base server 101 can be used to store structured, semi-structured, or unstructured data such as exercise prescriptions, rehabilitation guidelines, nutritional advice, exercise risk warnings, and exercise contraindications. For example, a first data entry can be obtained from the sports and health knowledge base server 101, and a second data entry that meets preset semantic matching conditions with the first data entry can also be obtained from the same server.

[0118] The vector database server 102 can be used to store the entry vectors corresponding to each data entry in the sports and health knowledge base, and provide semantic retrieval services based on the entry vectors. For example, each data entry in the sports and health knowledge base can be converted into an entry vector using an embedding model, and each data entry can be associated with and stored in the vector database server 102. Then, based on the first entry vector corresponding to the first data entry, a semantically similar candidate entry vector can be retrieved using an approximate nearest neighbor search algorithm, and the corresponding data entry can be determined as the second data entry.

[0119] The large language model server 103 can be used to deploy the large language model under test, receive test questions and various types of test contexts, and generate corresponding test answers for each test question. The large language model under test can be a large language model that is intended to be applied to the field of sports and health and has not yet been launched or is in the pre-launch verification stage, or it can be a large language model that has been launched but needs to be re-evaluated.

[0120] The large language model evaluation server 104 can be used to deploy the evaluation model and evaluate the large language model under test based on each test answer and the standard answer. When an updated answer corresponding to a third data entry is determined, the large language model evaluation server 104 can also evaluate the test answer corresponding to the test context of the data modification type, in conjunction with the updated answer. For example, the large language model under test server 103 and the large language model evaluation server 104 can be configured on different servers to reduce the mutual influence between the test answer generation process and the evaluation process.

[0121] The client terminal 105 can be used to receive model evaluation instructions initiated by testers, and display the evaluation progress, each answer to be tested, the evaluation results, or the comparison results between multiple large language models to be tested. The client terminal 105 can be, but is not limited to, a smartphone, desktop computer, laptop computer, or tablet computer. It should be noted that the above-mentioned servers can be set up separately, or at least two of them can be integrated into the same computer device according to actual deployment needs; the data selection, question construction, test context construction, model invocation, and result statistics in the model evaluation method can be completed collaboratively by the client terminal 105, any server, or multiple communicating computer devices.

[0122] For reference Figure 3 , Figure 3 This is a schematic diagram illustrating the overall processing flow of a model evaluation method provided in an embodiment of this application. Figure 3 As shown, the overall processing flow may include stages such as constructing a vector database 301, retrieving similar data 302, constructing a test set 303, performing three types of context tests 304, and performing fidelity evaluation 305.

[0123] In the vector database construction stage 301, data entries can be extracted from the sports and health knowledge base, and each data entry can be converted into a corresponding entry vector through an embedding model to construct the vector database. In the similar data retrieval stage 302, the data entry to be evaluated can be used as the first data entry, and based on the first entry vector corresponding to the first data entry, the top K data entries with high semantic similarity can be retrieved from the vector database to obtain at least one second data entry. The second data entry can be a real data entry in the sports and health knowledge base, and has the same or similar sports and health theme as the first data entry, but cannot be used alone to uniquely determine the standard answer to the test question. Therefore, it supports targeted fine-grained testing of individual data entries, and can accurately locate risky entries in the sports and health knowledge base that are prone to model fabrication errors.

[0124] In the test set construction phase 303, test questions and corresponding standard answers can be constructed based on the first data entries, and modified according to the first data entries to obtain the third data entries; the standard answers can also be updated according to the third data entries to obtain updated answers. In the three-context testing phase 304, test contexts for data integrity types can be constructed based on the first and second data entries, test contexts for data missing types can be constructed based on the second data entries, and test contexts for data modification types can be constructed based on the second and third data entries; subsequently, the same test question is input into the large language model under test with the above-mentioned test contexts of different types to obtain the corresponding test answers.

[0125] In the fidelity assessment phase 305, an evaluation model can be used to assess each test answer. Specifically, for test contexts with complete data, the consistency between the test answer and the standard answer can be used to assess the test language model's ability to respond when complete data is available. For test contexts with missing data, the ability to correctly refuse to answer when current data is insufficient can be assessed. For test contexts with modified data, the consistency between the test answer and the updated answer can be used to assess whether the test language model faithfully uses the currently provided third data item, rather than relying on existing memories formed during training. This allows for a complete application process, from real data in the sports and health knowledge base to test data construction, execution in different test scenarios, and output of evaluation results.

[0126] To facilitate understanding, the above model evaluation method is illustrated below using a sports rehabilitation scenario. This example is only used to illustrate the relationship between the various data points and is not intended to limit the scope of protection of this application.

[0127] Suppose the first data entry in the sports and health knowledge base is: "Post-knee surgery patients should begin straight leg raise training in the second week after surgery, 10 repetitions per set, 3 sets per day." This first data entry contains three key information elements: exercise type (straight leg raise training), start time (second week after surgery), and training volume (10 repetitions per set, 3 sets per day). First, the first data entry is converted into a 512-dimensional vector feature using an embedding model and stored in a vector database. Next, an approximate nearest neighbor search algorithm is used to retrieve the two data entries with the highest cosine similarity to the first entry vector in the sports and health knowledge base, obtaining two truly similar data entries that are semantically related to "post-knee surgery rehabilitation" but differ in specific details, as the second data entry. For example, the second data entry could be: "Patients after knee surgery should begin isometric quadriceps contraction exercises in the third week post-surgery, holding each contraction for 5 seconds, 4 sets daily." and "Patients after knee surgery should begin ankle pump exercises in the first week post-surgery, 20 repetitions per set, 5 sets daily." While both the first and second data entries relate to post-knee rehabilitation, they cannot be used to definitively determine the start time and number of repetitions for straight leg raise exercises. Furthermore, these similar second data entries all originate from the sports and health knowledge base itself and are actual data entries existing within the database.

[0128] Then, a test question can be constructed based on the first data entry: "Based on the given data, in which week post-surgery should the patient begin straight leg raise training? How many repetitions per set? How many sets per day?" The standard answer is: "Starting in the second week post-surgery, 10 repetitions per set, 3 sets per day." The key information in this question (straight leg raise training, second week, 10 repetitions per set, 3 sets per day) only appears explicitly in the first data entry and is not uniquely certain in the second data entry and the model's training knowledge. Therefore, the model must faithfully adhere to the original data in the context to answer correctly.

[0129] Modifying the model allows for semantic perturbation and modification of key information in the first data entry. For example, "week 2 post-surgery" can be changed to "week 3 post-surgery," and "10 repetitions per set" can be changed to "15 repetitions per set," resulting in the third data entry: "Patients after knee surgery should begin straight leg raise training in week 3 post-surgery, 15 repetitions per set, 3 sets per day." Based on this, the standard answer can be updated to: "Starting in week 3 post-surgery, 15 repetitions per set, 3 sets per day."

[0130] The test context for data integrity type can include a first data entry and the two second data entries mentioned above. In this test context, the large language model under test should output a test answer consistent with the standard answer. If the test answer is inconsistent with the standard answer, it indicates that even with the complete data used to determine the answer, the large language model under test failed to accurately identify the key information in the first data entry.

[0131] For test contexts with missing data, only the two second data entries mentioned above may be included, excluding the first data entry. In this test context, the large language model under test should indicate that it cannot determine the start time and number of training iterations for straight leg raises based on the given data. If the large language model still outputs a standard answer or other specific answer, it indicates that it generated unfounded content given the current lack of data.

[0132] The test context for data modification types can include the third data entry and the two second data entries mentioned above, but not the first data entry. The large language model under test should output the test answer consistent with the updated answer within this test context. If the large language model under test still outputs the standard answer before modification, it indicates that it has not faithfully used the currently provided third data entry.

[0133] The evaluation model can judge the three test answers separately and associate the judgment results with the first data entry. After repeatedly processing multiple first data entries in the sports and health knowledge base, it can statistically analyze the correctness rate of complete answers, the correct rejection rate of missing data, and the correctness rate of modified data, and determine the risk status of each first data entry. Thus, it can achieve a complete process from data entries in the sports and health knowledge base to automatically constructing test cases, forming different test scenarios, automatically generating test answers, automatically evaluating, and locating high-risk data entries.

[0134] The above describes some specific implementations of the model evaluation method provided in the embodiments of this application. Based on this, the present application also provides a corresponding apparatus. The apparatus provided in the embodiments of this application will be described below from the perspective of functional modularity.

[0135] See Figure 4 The figure is a schematic diagram of the structure of a model evaluation device 400 provided in an embodiment of this application. The device 400 may include: The acquisition module 401 is used to acquire a first data entry and a second data entry from the sports and health knowledge base, wherein the second data entry and the first data entry satisfy a preset semantic matching condition; The question construction module 402 is used to construct test questions and corresponding standard answers based on the first data entry; The test scenario construction module 403 is used to construct test contexts of different types according to the first data entry, the second data entry, and the third data entry, wherein the third data entry is obtained by modifying the first data entry. The answer module 404 is used to determine each answer to be tested corresponding to the test question based on the various test contexts and the test question using the large language model to be tested. Evaluation module 405 is used to evaluate the large language model to be tested based on each answer to be tested and the standard answer, and obtain the evaluation result.

[0136] Optionally, the various types of test contexts include test contexts for data integrity, data missing, and data modification. The test scenario construction module 403 is specifically used for: Based on the first data entry and the second data entry, a test context of the data completeness type is constructed. The test context of the data completeness type is used to test the response capability of the large language model under test in the case of complete data. Based on the second data entry, a test context for the data missing type is constructed, which is used to test the response ability of the large language model under test in the case of data missing. Based on the second data entry and the third data entry, a test context of the data modification type is constructed. The test context of the data modification type is used to test the response ability of the large language model under test when the data is modified.

[0137] Optionally, the method for determining the third data entry includes the following components: The modification unit is used to modify the key information in the first data entry by modifying the model to obtain the third data entry; Correspondingly, the device further includes: An update module is used to update the standard answer based on the third data entry to obtain an updated answer, which is used to evaluate the large language model to be tested.

[0138] Optionally, the evaluation module 405 is specifically used for: The large language model under test is evaluated by evaluating the model based on the individual test answers, the standard answer, and the updated answer, and the evaluation result is obtained.

[0139] Optionally, the device further includes: The output module is used to determine, for multiple different large language models under test, the corresponding test answers output by the large language models under test for the test questions based on the test contexts of each different type and the test questions; The model evaluation module is used to evaluate the large language model to be tested for multiple different models based on the individual test answers and the standard answers, and obtain the evaluation results.

[0140] Optionally, the method for determining the second data entry includes the following elements: The retrieval unit is used to retrieve, through a preset search algorithm, a data entry in the knowledge base that meets a preset semantic matching condition with the first data entry, and use it as the second data entry.

[0141] Optionally, the problem construct modulus 402 is specifically used for: By constructing a problem model, based on the first data entry, test questions and corresponding standard answers are constructed, and the standard answers include the content in the first data entry.

[0142] This application also provides corresponding devices and computer storage media for implementing the solutions provided in this application.

[0143] The device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to enable the device to perform the model evaluation method described in any embodiment of this application.

[0144] The computer storage medium stores a computer program. When the code is run, the device running the computer program implements the model evaluation method described in any embodiment of this application.

[0145] In the embodiments of this application, the terms "first" and "second" (if they exist) are used only as name identifiers and do not represent the order of first and second.

[0146] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a general-purpose hardware platform. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0147] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. The components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment solution according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0148] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model evaluation method, characterized in that, The method includes: Obtain the first data entry and the second data entry from the sports and health knowledge base, wherein the second data entry and the first data entry satisfy a preset semantic matching condition; Based on the first data entry, construct test questions and corresponding standard answers; Based on the first data entry, the second data entry, and the third data entry, test contexts of different types are constructed respectively, wherein the third data entry is obtained by modifying the first data entry; Using the large language model to be tested, based on the various test contexts and the test questions, determine the corresponding answers to the test questions; The large language model to be tested is evaluated based on the individual test answers and the standard answers to obtain the evaluation results.

2. The method according to claim 1, characterized in that, The various test context types include test contexts for data integrity, missing data, and modified data. The construction of each different type of test context based on the first data entry, the second data entry, and the third data entry includes: Based on the first data entry and the second data entry, a test context of the data completeness type is constructed. The test context of the data completeness type is used to test the response capability of the large language model under test in the case of complete data. Based on the second data entry, a test context for the data missing type is constructed, which is used to test the response ability of the large language model under test in the case of data missing. Based on the second data entry and the third data entry, a test context of the data modification type is constructed. The test context of the data modification type is used to test the response capability of the large language model under test when the data is modified.

3. The method according to claim 1, characterized in that, The method for determining the third data entry includes: By modifying the model, the key information in the first data entry is modified to obtain the third data entry; The method further includes: The standard answer is updated based on the third data entry to obtain an updated answer, which is used to evaluate the large language model under test.

4. The method according to claim 3, characterized in that, The evaluation of the large language model to be tested, based on each answer to be tested and the standard answer, yields an evaluation result, including: The large language model under test is evaluated by evaluating the model based on the individual test answers, the standard answer, and the updated answer, and the evaluation result is obtained.

5. The method according to claim 1, characterized in that, The method further includes: For multiple different large language models to be tested, based on the various test contexts and test questions of the large language models to be tested, the corresponding test answers output by the large language models to be tested for the test questions are determined. For multiple different large language models to be tested, the large language models to be tested are evaluated based on the individual test answers and the standard answers to obtain the evaluation results.

6. The method according to claim 1, characterized in that, The methods for determining the second data entry include: Using a preset search algorithm, a data entry that meets the preset semantic matching conditions with the first data entry is retrieved from the knowledge base and used as the second data entry.

7. The method according to claim 1, characterized in that, The step of constructing test questions and corresponding standard answers based on the first data entry includes: By constructing a problem model, based on the first data entry, test questions and corresponding standard answers are constructed, and the standard answers include the content in the first data entry.

8. A model evaluation device, characterized in that, The device includes: The acquisition module is used to acquire a first data entry and a second data entry from the sports and health knowledge base, wherein the second data entry and the first data entry satisfy a preset semantic matching condition; The question construction module is used to construct test questions and corresponding standard answers based on the first data entry; The test scenario construction module is used to construct test contexts of different types based on the first data entry, the second data entry, and the third data entry, wherein the third data entry is obtained by modifying the first data entry. The answer module is used to determine the corresponding answers to the test questions based on the various test contexts and the test questions, using the large language model to be tested. The evaluation module is used to evaluate the large language model to be tested based on each answer to be tested and the standard answer, and obtain the evaluation result.

9. A model evaluation device, characterized in that, The device includes: Memory, used to store computer programs; A processor for executing the computer program to cause the device to perform the model evaluation method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, implements the model evaluation method as described in any one of claims 1 to 7.

Images