Large-model long text processing capability evaluation method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a large model long text processing capability evaluation method, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, and can more accurately reflect the actual long text processing capability of a large model. The large model long text processing capability evaluation method comprises the steps that a first evaluation task is obtained, the first evaluation task comprises a first task sample and a first task problem, and the first task sample comprises first sample content and a first sample result corresponding to the first sample content; the first task problem is a problem for obtaining a corresponding result for the second sample content; obtaining an interference text; inserting the first task sample into an interference text, and combining the interference text inserted with the first task sample and the first task problem to obtain a task text; executing a first test, the first test comprising: inputting the task text into the large model to obtain a first answer output by the large model based on the first task question; and obtaining an evaluation result of the large model according to the first answer.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a method for evaluating the processing capability of large-scale models and long texts, an electronic device, and a storage medium. Background Art

[0002] With the development of artificial intelligence, the performance evaluation of large language models (LLM) is becoming more and more important. In order to more effectively evaluate the processing ability of large models for long texts, the current evaluation method can evaluate the processing ability of long texts by performing needle-in-a-haystack retrieval tasks on long texts. However, the evaluation results obtained in this way are still different from the performance of large models in actual long text processing tasks, resulting in low practical value of the evaluation results. Summary of the invention

[0003] The embodiment of the present application provides a method for evaluating the long text processing capability of a large model, and the electronic device and storage medium can more accurately reflect the actual long text processing capability of the large model.

[0004] In a first aspect, an embodiment of the present application provides a method for evaluating the long text processing capability of a large model, comprising: obtaining a first evaluation task, the first evaluation task comprising a first task example and a first task question, the first task example comprising a first example content and a first example result corresponding to the first example content, the first task question being a question for obtaining a corresponding result for a second example content, and the second example content being different from the first example content; obtaining interference text; inserting the first task example into the interference text, and combining the interference text into which the first task example is inserted and the first task question to obtain a task text; executing a first test, the first test comprising: inputting the task text into a large model to obtain a first answer output by the large model based on the first task question; and obtaining an evaluation result of the large model based on the first answer.

[0005] By inserting task samples into interference text and combining them with task questions to obtain task text, the task text is input into the large model for testing. By using a needle-in-a-haystack task setting method, the test is closer to the complex scenarios in real applications, thereby more accurately reflecting the actual long text processing capabilities of the large model and improving the practical value of the evaluation results.

[0006] In a possible implementation, before obtaining the first evaluation task, the method also includes: obtaining multiple test samples; replacing semantic tags in the multiple test samples with preset tags; at least part of the multiple test samples are first task samples, at least one of the multiple test samples is a first task problem, and the first task problem is different from the first task sample.

[0007] By replacing the labels in regular test examples with meaningless labels, that is, labels that are not related to the content, the possibility of data leakage can be reduced and the learning ability of the model can be more accurately reflected. Because if the labels are related to the content, the large model may be able to answer the correct answer without learning because the content itself is a training sample for the large model, and the question can be answered using prior knowledge. For the first evaluation task after replacing the labels with labels that are not related to the content, the large model is forced to retrieve and learn the first task examples embedded in the long text in order to correctly answer the first task question, which can more accurately reflect the long text processing performance of the large model.

[0008] In one possible implementation, before obtaining the first evaluation task, the method also includes: performing a pre-test, the pre-test including multiple second tests based on the i-th number of second task samples, the value of i is 1, 2, ..., n, and n is an integer greater than 1; each second test includes: inputting the i-th number of second task samples and at least one second task question into the large model, and obtaining a second answer output by the large model based on the at least one second task question; the i-th number of second task samples and at least one second task question are part of multiple test samples, and the i-th number of second task samples are different in any different second tests under the same i value; obtaining the accuracy of the second answers of the multiple second tests of the i-th number of second task samples; the i value corresponding to the highest accuracy of the second answers of the multiple second tests of the i-th number of second task samples is a, and the a-th number is the number of first task samples in the first evaluation task.

[0009] The pre-test is used to determine the number of first task samples in the first evaluation task, so that the subsequent first test can be tested based on the first task samples determined by the pre-test to improve the evaluation accuracy. Especially for the ICL task, the number of task samples input into the large model during the test has different effects on the test results. Therefore, setting the most accurate number of task samples according to the most accurate test results can improve the evaluation accuracy. In addition, the first test focuses on evaluating the long text processing capability of the large model, while the pre-test focuses on evaluating other capabilities of the large model, such as ICL capabilities. By comparing the differences in the performance of the two evaluation results, the impact of long text input on the original ICL capability of the large model is evaluated, and it is observed whether the original reasoning ability of the large model is lost when processing long texts. In addition, through the introduction of the loss rate indicator, it can be used to accurately quantify the performance changes of the model on texts of different lengths to improve the interpretability and practical guidance of the evaluation results.

[0010] In one possible implementation, executing the first test includes executing multiple first tests; obtaining the evaluation result of the large model based on the first answer includes: calculating the depreciation rate, the depreciation rate is equal to (A2-A1) / A2, wherein A2 is the accuracy of the second answers of the multiple second tests of the a-th number of second task samples, and A1 is the accuracy of the first answers of the multiple first tests.

[0011] The loss rate can reflect the degree of performance degradation of a large model in a long text environment, and provides a more accurate quantitative standard for the long text processing capability of a large model. The introduction of the loss rate can make the performance comparison between different large models more objective and reliable.

[0012] In a possible implementation, in any different first tests, at least one of the content of the interference text, the position where the first task sample is inserted into the interference text, and the content of the first assessment task is different.

[0013] During the construction of the task text, the evaluation can be accurately quantified based on the different lengths and insertion positions of the interference text, so as to achieve a refined evaluation of the effective processing length of the large model, and in conjunction with the loss rate, the performance changes of the large model based on interference texts of different lengths can be accurately quantified and evaluated.

[0014] In a possible implementation, the test examples are context-based learning (ICL) task test examples.

[0015] In a possible implementation, the first evaluation task is a large-scale multi-task language understanding MMLU task.

[0016] In a possible implementation, before obtaining the first evaluation task, it also includes: executing multiple third tests, each third test includes: inputting the third task sample and the third task question into the large model, and obtaining the third answer output by the large model based on the third task question; the third task sample and the third task question are the samples and questions corresponding to the MMLU task; in any different third tests, the third task samples are different; obtaining the third test accuracy, and the third test accuracy is the accuracy of the third answers of multiple third tests; executing the first test includes executing multiple first tests; in any different first tests, at least one of the content of the interference text, the position where the first task sample is inserted into the interference text, and the content of the first evaluation task is different; obtaining the evaluation result of the large model according to the first answer includes: calculating the depreciation rate, the depreciation rate is equal to (A3-A1) / A3, wherein A3 is the third test accuracy, and A1 is the accuracy of the first answer of multiple first tests.

[0017] In a second aspect, an embodiment of the present application provides an electronic device, comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device executes the above method.

[0018] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium including a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart of a method for evaluating the processing capability of a large model long text in an embodiment of the present application; Figure 2 A flowchart of another large model long text processing capability evaluation method in an embodiment of the present application; Figure 3 A flowchart of another large model long text processing capability evaluation method in an embodiment of the present application; Figure 4 A flowchart of another large model long text processing capability evaluation method in an embodiment of the present application; Figure 5 A schematic diagram of a scenario-based flow chart of a large model long text processing capability evaluation method in an embodiment of the present application; Figure 6 This is a structural block diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0021] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0022] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0023] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0024] Before introducing the embodiments of the present application, the related technologies and their technical problems are first explained.

[0025] NeedleBench is an evaluation method in related technologies, including: Single-Needle Retrieval (S-RT), which inserts a key piece of information at a specific depth and requires the model to accurately locate and extract the information to test the model's ability to extract a single key piece of information in a long text; Multi-Needle Retrieval (M-RT), which scatters multiple pieces of relevant information at different positions in the text to evaluate the model's ability to extract multiple pieces of relevant information in a long text; Multi-Needle Reasoning (M-RS), which requires the model to extract multiple pieces of information from a long text and use the information to answer questions logically; Ancestral Trace Challenge (ATC), which builds a series of simple first-order logic reasoning to form an information chain and requires the model to fully understand and answer questions. Some of these tasks involve the evaluation of large models processing long texts, but only involve the ability to retrieve in long texts.

[0026] LongICLBench is a benchmark test set in related technologies for evaluating large language models for extreme label classification tasks in long context scenarios. The main features of this technical solution are as follows: the label classification task examines the model's ability to accurately classify in a large-scale label space; the task has a large range of context lengths, which aims to test the model's understanding and reasoning ability when processing large amounts of information; the In-Context Learning (ICL) paradigm is adopted, requiring the model to understand the task and make predictions through given examples; it has a progressive difficulty design to comprehensively evaluate the model's ability boundaries. It can be seen that LongICLBench is an evaluation based on the overall scenario of long texts. Due to the failure to fully decouple the long text processing capability from the other capabilities of the model, for example, it is difficult to distinguish whether the difference in the evaluation results is due to the long text processing capability or the ICL capability itself, which leads to deviations in the evaluation results.

[0027] RULER is an evaluation method in related technologies, which aims to comprehensively evaluate the long text processing capabilities of large language models. The whole process can be divided into three main stages: Retrieval stage: The system first generates a large number of long texts containing specific information. These texts cover different lengths and complexities, simulating various scenarios in the real world.

[0028] Aggregation / Inference / Matching Phase: In this core phase, the system performs in-depth processing on the retrieved text. This includes information aggregation, logical reasoning, and matching of key information. This phase tests the model's ability to understand, reason, and integrate information.

[0029] Output phase: Finally, the system requires the model to generate a response based on the results of the previous two phases. This phase evaluates the model's ability to transform complex information into meaningful output.

[0030] These three stages form a closed loop that allows the system to repeatedly test the model’s performance on different tasks and text lengths.

[0031] In the related technologies, the evaluation methods for large models’ ability to process long texts have relatively simple task settings, which are not highly relevant to complex actual application scenarios. Therefore, the unrealistic task design causes the evaluation results to be unable to truly reflect the performance of the model in actual long text processing, resulting in low practical value of the evaluation results.

[0032] The embodiments of the present application can more accurately reflect the actual long text processing capabilities of the large model. The embodiments of the present application are described below.

[0033] like Figure 1 As shown, the embodiment of the present application provides a method for evaluating the processing capability of a large model long text, including: Step 101: Obtain a first evaluation task, where the first evaluation task includes a first task example and a first task question.

[0034] Among them, the first task example is an example used for large model learning, and the first task question is a question related to the first task example, which is used to enable the large model to answer based on the learning of the first task example. The first task example includes the first example content and the first example result corresponding to the first example content. The first task question is a question that obtains a corresponding result for the second example content, and the second example content is different from the first example content. For example, the first example task is a classification task, the first example content is the input content, the first example result is the classification result corresponding to the input content, and the second example content is another input content. In the subsequent evaluation process, the large model is required to learn a small amount of input content and its corresponding classification results, and then obtain the corresponding classification results based on the other input content, so as to facilitate evaluation based on the final classification results.

[0035] For example, the first evaluation task is: ["This is a sentence classification task. Given a sentence, classify it as 'cat' or 'apple'.\nSentence: "I doubt this person would have a hard time sitting through this." Classification: apple. Sentence: "Interesting things" Classification: cat. Sentence: "People who think acting is the heart and soul of a movie" Classification:\nAnswer:"].

[0036] The sentence [sentence: "I doubt this person would have a hard time sitting through this." Category: Apple. ] is the first task example A, the sentence [sentence: "Interesting things" Category: Cat. ] is the first task example B, and the sentence [sentence: "People who think that acting is the heart and soul of a movie" Category: \nAnswer:"] is the first task question. The correct answer to the first task question is "Cat".

[0037] It should be noted that, in this example, the first evaluation task is a classification task, but there is no logical correlation between the sentence and the corresponding classification result. Even so, the irrelevant content can be classified according to the preset requirements. The specific reasons and the method of obtaining such an evaluation task will be introduced in detail in the subsequent content. It can be understood that in other embodiments, the classification result in the first evaluation task and the sentence can also have a logical correlation.

[0038] In the first task example A, [sentence: "I doubt that this person will have a hard time sitting through this."] is the first example content A1, and [category: apple] is the first example result A2 corresponding to the first example content A1, that is, the classification result.

[0039] In the first task example B, [sentence: "interesting things"] is the first example content B1, and [category: cat] is the first example result B2 corresponding to the first example content B1, that is, the classification result.

[0040] In the first task problem, [sentence: "People who think that acting is the core and soul of film"] is the second sample content, and the first task problem requires giving the classification result corresponding to this sentence, that is, the answer.

[0041] The number of first task examples and the number of first task questions in the first evaluation task are not limited in the embodiments of the present application. In this example, the first evaluation task has two first task examples and one first task question. In other possible implementations, the first evaluation task may have only one first task example, or there may be a greater number of first task examples, and the first evaluation task may also have a greater number of first task questions. However, for the evaluation of large models, obtaining an answer based on a question each time is more conducive to the overall evaluation process design such as data statistics. It should be noted that the specific example of the first evaluation task here is an ICL task. The embodiments of the present application do not limit the specific type of the first evaluation task. For example, the first evaluation task may also be a Massive Multitask Language Understanding (MMLU) task, which will be described in detail later.

[0042] Step 102: Obtain interference text.

[0043] Interference text is irrelevant or low-relevant content used to increase the difficulty of the model. In real scenarios, the long texts that large models need to process often contain a lot of irrelevant information. The presence of interference text helps simulate the situation in the real environment. For example, part of the content in a large-scale long text dataset can be selected as interference text. The dataset can be composed of a large number of books, covering various types of literary works, such as novels, poems, non-fiction articles, etc., providing rich and diverse text content. The interference text selected in the dataset can provide a large amount of diverse background information for the evaluation, which is conducive to testing the ability of large models to locate and utilize information in irrelevant information.

[0044] Step 103: insert the first task example into the interference text, and combine the interference text with the first task example inserted and the first task question to obtain the task text.

[0045] For example, the task text is: ["input": "This is a very long context: {Preceding content of distracting text} \nThis is a sentence classification task. Given a sentence, classify it as 'cat' or 'apple'. \nSentence: "I doubt this person would have a hard time sitting through this." Classification: apple. Sentence: "Interesting things" Classification: cat.

[0046] {Poster content that interferes with the text}. \nBased on the content of the book, question: Sentence: "People who believe that acting is the heart and soul of film" Category: \nAnswer: ”].

[0047] The first task example is inserted between the preceding content and the following content of the interference text, and the first task question is set after the interference text, thus obtaining the task text.

[0048] Step 104: Execute a first test, which includes: inputting the task text into the large model to obtain a first answer output by the large model based on the first task question.

[0049] In the task text, the long distracting text is the "sea". During the first test, the large model needs to retrieve the first task examples located in it while processing a large amount of irrelevant text, that is, the first example content and its corresponding first example results, and use the learning of the first task examples to answer the first task question at the end of the text, that is, answer the sentence: "People who think that acting is the core and soul of the movie" The classification answer is cat or apple, the correct answer is cat, and the accuracy of the large model's answer can be judged based on the correct answer. This needle-in-the-haystack task setting method is closer to the complex scenarios in real applications. For example, the large model is required to perform contextual learning based on several classification questions inserted in the distracting text, and then answer a new classification question; or, the large model is required to perform contextual learning based on several knowledge questions inserted in the distracting text, and then answer a new knowledge question.

[0050] Step 105: Obtain the evaluation result of the large model according to the first answer.

[0051] Whether the answer output by the large model for the first task question is correct can be determined based on the correct answer, so as to determine the long text processing performance of the large model. For example, in step 104, multiple first tests can be performed, and the task texts can be different in different first tests. Then, in step 105, the accuracy of the answers of the large model in multiple first tests is determined to achieve the evaluation of the large model.

[0052] The large model long text processing capability evaluation method of the embodiment of the present application inserts task samples into interference text, obtains task text by combining task problems, inputs the task text into the large model for testing, and utilizes a needle-in-a-haystack task setting method to make the test closer to the complex scenarios in real applications, thereby more accurately reflecting the actual long text processing capability of the large model and improving the practical value of the evaluation results.

[0053] In some embodiments, Figure 2 As shown, before obtaining the first evaluation task, it also includes: Step 201: Obtain multiple test samples; The test sample may be, for example, a sentiment classification data set, including corresponding labels and content. For example, three test samples are shown below: [Sentence: "I doubt this person would have a hard time sitting through this." Category: Negative.

[0054] Sentence: "Interesting things" Category: Positive.

[0055] Sentence: "someone who believes that acting is the heart and soul of cinema" Category: Positive].

[0056] Among them, the sentence is the content, the classification result is the label, and the sentence and the corresponding classification result are a test example.

[0057] Step 202: Replace the semantic tags in multiple test samples with preset tags; For example, for the three test samples obtained in step 201, the original input text is retained, but the semantic tags are replaced with preset tags, which are common words irrelevant to the content of the test samples. For example, "positive" is replaced with "cat", and "negative" is replaced with "apple". The new tags after replacement maintain a one-to-one correspondence with the input text, but there is no obvious semantic connection. The three test samples after the tag replacement are: [Sentence: "I doubt this person would have a hard time sitting through this." Category: Apple.

[0058] Sentence: "Interesting things" Category: Cats.

[0059] Sentence: "someone who believes that acting is the heart and soul of cinema" Category: Cats].

[0060] Then, the above step 101 can be performed, and the first evaluation task in the above step 101 can be constructed according to the three test examples after replacing the labels. The first evaluation task can be organized into the evaluation format of few-shot learning. At least part of the multiple test examples are first task examples, for example, the two first task examples in the first evaluation task are the first two of the three test examples [sentence: "I suspect that this person will have a hard time sitting through this." Category: Apple. Sentence: "Interesting things" Category: Cat. ]. At least one of the multiple test examples is a first task problem, and the first task problem is different from the first task example. For example, the first task problem is a test example to be classified [sentence: "People who think that acting is the core and soul of a movie"], and the classification of the original test example is removed, which can be used as the first task problem.

[0061] It should be noted that only three test samples are used to construct the first evaluation task as an example here. In step 201, a larger number of test samples can be obtained. In step 101, a small number of test samples are selected to construct the first evaluation task.

[0062] By replacing the labels in regular test examples with meaningless labels, that is, labels that are not related to the content, the possibility of data leakage can be reduced and the learning ability of the model can be more accurately reflected. Because if the labels are related to the content, the large model may be able to answer the correct answer without learning because the content itself is a training sample for the large model, and the question can be answered using prior knowledge. For the first evaluation task after replacing the labels with labels that are not related to the content, the large model is forced to retrieve and learn the first task examples embedded in the long text in order to correctly answer the first task question, which can more accurately reflect the long text processing performance of the large model.

[0063] In some embodiments, Figure 3 As shown, before step 101, obtaining the first evaluation task, the following is also included: Step 301: perform a pre-test, where the pre-test includes multiple second tests based on an i-th number of second task samples, where i is 1, 2, ..., n, and n is an integer greater than 1; Each second test includes: inputting the i-th number of second task examples and at least one second task question into the large model, and obtaining a second answer output by the large model based on the at least one second task question.

[0064] For example, assume that 5000 test samples are obtained in step 201, and all labels therein are replaced with preset labels in step 202. In step 301, assume that n=3, that is, the values ​​of i are 1, 2, and 3 respectively, and assume that the first number is 1, the second number is 2, and the third number is 3.

[0065] The i-th number of second task examples and at least one second task question are part of multiple test examples, and the i-th number of second task examples are different in any different second tests under the same i value.

[0066] When the value of i is 1, the second test is performed, for example, 100 times. In each second test, a second task sample is selected from 5000 test samples, a second task problem is selected, and the selected second task sample and the second task problem are input into the large model, so that the second answer output by the large model based on the second task problem can be obtained, and whether the second answer output by the large model is accurate can be judged. In different second tests, the selected second task samples and second task problems are different. It can be selected according to the preset rules. For example, in the first second test, the first test sample is used as the second task sample, and the second test sample is used as the second task problem. In the second second test, the third test sample is used as the second task sample, and the fourth test sample is used as the second task problem, and so on. It is understandable that random selection may also be performed, for example, in the first test, the 245th test example is randomly selected as the second task example, the 2020th test example is randomly selected as the second task problem, and in the second test, the 3424th test example is randomly selected as the second task example, and the 601st test example is randomly selected as the second task problem. Regardless of how the selection is performed, only one second task example is selected.

[0067] When the value of i is 2, the second test is performed, for example, 100 times. In each second test, two second task samples are selected from 5000 test samples, and a second task problem is selected. The selected second task samples and the second task problem are input into the large model, and the second answer output by the large model based on the second task problem can be obtained, and whether the second answer output by the large model is accurate can be judged. In different second tests, the selected second task samples and second task problems are different. It can be selected according to the preset rules. For example, in the first second test, the first and second test samples are used as the two second task samples, and the third test sample is used as the second task problem. In the second second test, the fourth and fifth test samples are used as the two second task samples, and the sixth test sample is used as the second task problem, and so on. It is understandable that random selection may also be performed, for example, in the first test, the 245th and 2008th test examples are randomly selected as the two second task examples, and the 2020th test example is randomly selected as the second task problem; in the second test, the 3424th and 4499th test examples are randomly selected as the two second task examples, and the 601st test example is randomly selected as the second task problem. Regardless of how the selection is performed, only two second task examples are selected.

[0068] When the value of i is 3, the second test is performed, for example, 100 times. In each second test, three second task samples are selected from 5000 test samples, and a second task problem is selected. The selected second task samples and the second task problem are input into the large model, and the second answer output by the large model based on the second task problem can be obtained, and whether the second answer output by the large model is accurate can be judged. In different second tests, the selected second task samples and second task problems are different. It can be selected according to the preset rules. For example, in the first second test, the first, second and third test samples are used as the three second task samples, and the fourth test sample is used as the second task problem. In the second second test, the fifth, sixth and seventh test samples are used as the three second task samples, and the eighth test sample is used as the second task problem, and so on. It is understandable that random selection may also be performed, for example, in the first test, the 245th, 2008th and 3224th test examples are randomly selected as the three second task examples, the 2020th test example is randomly selected as the second task problem, and in the second test, the 3424th, 4499th and 765th test examples are randomly selected as the three second task examples, and the 601st test example is randomly selected as the second task problem. Regardless of how the selection is performed, only three second task examples are selected.

[0069] That is to say, in the pre-test, the large model is tested based on different numbers of second task sample selections.

[0070] Step 302: Obtain the accuracy of the second answers of the second test of the i-th number of second task examples. After step 302, the above steps 101 to 105 are performed, and the specific process is not repeated here.

[0071] Among them, the i value corresponding to the highest accuracy of the second answers of multiple second tests of the i-th number of second task examples is a, and the a-th number is the number of first task examples in the first evaluation task.

[0072] For example, when the value of i is 1, the accuracy of the second answer of the second test, for example, 100 times, is 86%; when the value of i is 2, the accuracy of the second answer of the second test, for example, 100 times, is 92%; when the value of i is 3, the accuracy of the second answer of the second test, for example, 100 times, is 85%. The i value corresponding to the highest accuracy is 2, which means that the number of first task examples in the first evaluation task can be determined to be 2. That is, in the subsequent first test, the applied first evaluation task has two second task examples.

[0073] That is to say, the pre-test is used to determine the number of first task samples in the first evaluation task, so that the subsequent first test is based on the first task samples determined by the pre-test to improve the evaluation accuracy. Especially for the ICL task, the number of task samples input into the large model during the test has different effects on the test results. Therefore, setting the most accurate number of task samples according to the most accurate test results can improve the evaluation accuracy. In addition, the first test focuses on evaluating the long text processing capability of the large model, while the pre-test focuses on evaluating other capabilities of the large model, such as ICL capabilities. By comparing the differences in the performance of the two evaluation results, the impact of long text input on the original ICL capability of the large model is evaluated, and it is observed whether the original reasoning ability of the large model is lost when processing long texts. In addition, through the introduction of the loss rate indicator, it can be used to accurately quantify the performance changes of the model on texts of different lengths to improve the interpretability and practical guidance of the evaluation results.

[0074] In some embodiments, the above step 101, executing the first test, includes executing the first test multiple times.

[0075] The above-mentioned step 105, obtaining the evaluation result of the large model according to the first answer, includes: calculating the depreciation rate, the depreciation rate is equal to (A2-A1) / A2, wherein A2 is the accuracy of the second answer of multiple second tests of the a-th number of second task samples, and A1 is the accuracy of the first answer of multiple first tests.

[0076] The loss rate can reflect the degree of performance degradation of a large model in a long text environment, and provides a more accurate quantitative standard for the long text processing capability of a large model. The introduction of the loss rate can make the performance comparison between different large models more objective and reliable.

[0077] In addition, in addition to calculating the loss rate, the above step 105, obtaining the evaluation results of the large model according to the first answer, can also include: creating a loss rate matrix, for example, the rows of the matrix represent the length of the interference text, and the columns of the matrix represent the position where the first task sample is inserted into the interference text; using the loss rate matrix to draw a heat map to intuitively show the impact of length and insertion position on performance; calculating and drawing a curve of the average loss rate versus text length; calculating and drawing a curve of the average loss rate versus insertion position depth. The range of the loss rate is [0, 1]. The lower the loss rate, the closer the large model's ability to process long texts is to its performance in short texts, that is, the stronger its long text processing ability is.

[0078] In some embodiments, in any different first tests, at least one of the content of the interference text, the position where the first task sample is inserted into the interference text, and the content of the first assessment task is different.

[0079] Specifically, for example, in each first test, a length of content is randomly selected from the long text data set as the interference text of this test, wherein, due to the randomness of the selection, the content of the interference text is not. In addition, the interference text can be selected according to a certain interference text length range rule, for example, the length of the interference text of each first test is set to be selected from the range of 10000 to 100000 word tokens, and the selection step can be set to 10000 tokens. For example, in the first to the tenth first test, the length of the interference text is 10000 tokens, and in the 11th to the 20th first test, the length of the interference text is 20000 tokens, and so on. In addition, the position where the first task sample is inserted into the interference text can be set based on some preset depths, such as 0%, 25%, 50%, 75%, 100%. For example, in the first and second first tests, the insertion position of the first task sample is 0%, and in the third and fourth first tests, the insertion position of the first task sample is 25%, and so on. In each first test, according to the current test length L, a long text T is selected from the long text data set, and the insertion point P (the position after the d%th token in T) is determined according to the preset embedding depth parameter d, and the first task sample Q is inserted into the insertion point P of the interference text to form a task text T' containing the task. In the process of constructing the task text, the evaluation can be accurately quantified based on the different lengths and insertion positions of the interference text, so as to achieve a refined evaluation of the effective processing length of the large model, and to cooperate with the loss rate so that the performance changes of the large model based on interference texts of different lengths can be accurately quantified and evaluated.

[0080] In some embodiments, the test examples are contextual learning ICL task test examples, which can be used to evaluate the ability of the large language model to understand and perform new tasks through contextual examples. The ICL task test examples, combined with the replacement of meaningless labels, can more closely evaluate the contextual learning ability of the large model in a long text environment. When performing each first test, the large model is required to first locate the ICL task from the long text and then complete the ICL task.

[0081] In some embodiments, Figure 1 As shown, the first evaluation task may be an MMLU task, which is a benchmark test for evaluating the performance of language models in multiple fields and tasks.

[0082] For example, in step 101, a small number of samples can be selected from the MMLU evaluation data set to construct the first evaluation task. The MMLU evaluation data set can involve, for example, 57 subject areas, and 100 questions can be randomly selected from each area to form a subset of 5700 questions. Questions can be selected from the subset of 5700 questions to construct the first evaluation task. The computational complexity of the evaluation can be controlled while maintaining a wide coverage of the evaluation task.

[0083] For example, the first evaluation task is: [“Below are multiple-choice questions about physics (with answers).

[0084] What is the approximate speed of light in a vacuum? A. 3 × 10^5 m / s B. 3 × 10^6 m / s C. 3 × 10^7 m / s D. 3 × 10^8 m / s Answer: D What is Newton's first law of motion also called? A. Law of Conservation of MomentumB. Law of InertiaC. Law of Kinetic EnergyD. Law of Universal Gravitation Answer: B Which of the following is an example of an electromagnetic wave? A. Sound waves B. Earthquake waves C. Light waves D. Water waves Answer: C Which principle determines the magnitude of the buoyancy on an object? A. Law of momentum B. Torricelli's law C. Archimedean principle D. Law of conservation of enthalpy Answer: C Which of the following units is used to measure electric current? A. Volt (V) B. Ampere (A) C. Ohm (Ω) D. Joule (J) Answer: B In thermodynamics, what temperature corresponds to absolute zero? A. 0°C B. -100°C C. -273.15°C D. 100°C Answer:].

[0085] The first five questions with answers are examples of the first task, which are used for the large model to learn the context. The content before the answer is the content of the first example, and the answer options are the results of the first example. The last question without an answer is the first task question, which is used for the large model to answer. The correct answer to the first task question is "C". The large model can be evaluated based on whether its answer is consistent with the correct answer.

[0086] By inserting MMLU tasks into the interference text, a more realistic complex information environment can be simulated to evaluate the knowledge retrieval and knowledge application capabilities of the large model under the influence of long text. The selection of interference text can be the same as in the above embodiment, and will not be repeated here.

[0087] In some embodiments, Figure 4 As shown, before step 101, obtaining the first evaluation task, the following is also included: Step 401, perform multiple third tests, each of which includes: inputting the third task sample and the third task question into the large model, and obtaining a third answer output by the large model based on the third task question.

[0088] The third task examples and the third task questions are examples and questions corresponding to the MMLU task. In any different third tests, the third task examples are different.

[0089] Step 492: Obtain a third test accuracy rate, where the third test accuracy rate is the accuracy rate of the third answers to multiple third tests.

[0090] Executing the first test in the above step 104 includes executing the first test multiple times.

[0091] In any different first tests, at least one of the content of the interference text, the position where the first task sample is inserted into the interference text, and the content of the first evaluation task is different.

[0092] The above step 105, obtaining the evaluation result of the large model according to the first answer, includes: calculating the loss rate, the loss rate is equal to (A3-A1) / A3, wherein A3 is the accuracy of the third test, and A1 is the accuracy of the first answer of multiple first tests.

[0093] For example, 100 test cases are randomly selected from each of the 57 academic fields of the MMLU evaluation data set. In each third test, 5 questions are selected as 5 third task samples. Each third task sample also has a corresponding answer, and the third task question can also be selected from the MMLU evaluation data set. In multiple third tests, the large model can be evaluated using 100 test cases in each subject area. According to the answers output by the large model in the third test, the answers of the large model can be recorded, and the accuracy of each field can be calculated. Based on the third test, the evaluation of the knowledge application ability of the large model can be achieved.

[0094] The first task example Q can be constructed based on the same method by selecting questions from the MMLU evaluation data set. For example, in each first test, a length of content is randomly selected from the long text data set as the interference text of this test, wherein, due to the randomness of the selection, the content of the interference text is not. In addition, the interference text can be selected according to a certain interference text length range rule, for example, the length of the interference text of each first test is set to be selected from 10000 to 100000 word unit token range, and the selection step can be set to 10000 token, for example, in the 1st to 10th first test, the length of the interference text is 10000 token, and in the 11th to 20th first test, the length of the interference text is 20000 token, and so on. In addition, the position where the first task sample is inserted into the interference text can be set based on some preset depths, such as 0%, 25%, 50%, 75%, and 100%. For example, in the first and second first tests, the insertion position of the first task sample is 0%, and in the third and fourth first tests, the insertion position of the first task sample is 25%, and so on. In each first test, according to the current test length L, a long text T is selected from the long text data set, and the insertion point P (the position after the d%th token in T) is determined according to the preset embedding depth parameter d, and the first task sample Q is inserted into the insertion point P of the interference text to form a task text T' containing the task. In the construction process of the task text, the evaluation can be accurately quantified based on the different lengths and different insertion positions of the interference text to achieve a refined evaluation of the effective processing length of the large model, so as to cooperate with the depreciation rate so that the performance changes of the large model based on interference texts of different lengths can be accurately quantified and evaluated. When performing each first test, the large model is required to first locate the MMLU task from the long text, and then complete the MMLU task. During the multiple first tests, each of the 57 fields can be tested separately, with each field repeated 100 times, each time using a different long text and insertion position.

[0095] Based on the first test, the knowledge application ability of the large model in a long text environment can be evaluated. The difference between the two evaluation results can be used to evaluate the impact of long text input on the original knowledge level of the large model, and to observe whether the original knowledge application ability of the large model is lost when processing long text. In addition, by introducing the loss rate indicator, it can be used to accurately quantify the performance changes of the model on texts of different lengths, so as to improve the interpretability and practical guidance of the evaluation results.

[0096] In addition, in addition to calculating the loss rate, the above step 105, obtaining the evaluation results of the large model according to the first answer, can also include: creating a loss rate matrix, for example, the rows of the matrix represent the length of the interference text, and the columns of the matrix represent the position where the first task sample is inserted into the interference text; using the loss rate matrix to draw a heat map to intuitively show the impact of length and insertion position on performance; calculating and drawing a curve of the average loss rate versus text length; calculating and drawing a curve of the average loss rate versus insertion position depth. The range of the loss rate is [0, 1]. The lower the loss rate, the closer the large model's ability to process long texts is to its performance in short texts, that is, the stronger its long text processing ability is.

[0097] like Figure 5 As shown, the embodiment of the present application realizes a comprehensive evaluation of the long text processing ability of the large language model by constructing the ICL task and the MMLU task. The ICL task requires the model to complete a search in a long text and then perform a meaningless label few-shot learning process, while the MMLU task requires the model to complete a search and then perform a knowledge question-answering task. This design not only tests the information retrieval ability of the model, but also evaluates the model's learning and knowledge application capabilities in complex contexts.

[0098] The embodiment of the present application introduces the loss rate indicator to intuitively reflect the degree of degradation of the model's performance in a long text environment, and provides an accurate quantitative standard for the model's long text processing capability. This indicator makes the performance comparison between different models more objective and reliable. In terms of comprehensive evaluation, by combining the two tasks of ICL and MMLU, the contextual learning ability and knowledge application ability of the model are comprehensively evaluated. In terms of long text adaptability, the needle-in-a-haystack task design simulates the long text scenario in the real world and improves the practicality of the evaluation. In terms of quantitative indicators, the loss rate provides an intuitive and comparable indicator, which is convenient for horizontal comparison between different models. In terms of flexibility and scalability, the embodiment of the present application can be easily extended to other types of NLP tasks and has good versatility.

[0099] The embodiments of the present application can reveal the significant deficiencies of many large models claiming to have long window capabilities in actual applications. The experimental results are shown in Table 1.

[0100] Table 1 The needle-in-a-haystack ICL / MMLU loss rate multiplied by the percentage in Table 1 can be understood as the actual window length. The data in Table 1 reveals the performance differences of different models when processing long texts. The large model A performed the best, especially in the MMLU task, maintaining a high loss rate of 0.97, almost completely maintaining its short text processing ability. The loss rates of other models ranged from 0.6 to 0.78, indicating a loss of about 22%-40% in performance when processing 32k word length text. It is worth noting that the large model F has large performance differences in different tasks, and the large model G, despite claiming to have a 128k window length, still shows performance degradation in 32k length text processing. These results show the evaluation of some large models, emphasizing the importance of evaluating the actual long text processing capabilities of large models, and pointing out the challenges that current long window models may face in practical applications.

[0101] like Figure 6 As shown, an embodiment of the present application further provides an electronic device 900, which includes: a processor 901, and the processor 901 is used to execute a computer program or instruction in a memory 902 to implement a large model long text processing capability evaluation method as described in the above embodiment.

[0102] It should be noted that the processor 901 can be any chip with computing capabilities, and is not limited to a central processing unit (CPU). For example, the processor 901 can be a chip including one or more transistors, resistors, capacitors, and other circuit elements for implementing certain functions; or it can be an integrated circuit of various packages that can implement the above method.

[0103] Exemplarily, the processor 901 may include one or more processing units, for example: the processor 901 includes a neural-network processing unit (NPU), and may also include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a digital signal processor (DSP), a baseband processor, etc. Among them, different processing units can be independent devices or integrated in one or more processors. The controller can generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.

[0104] The memory 902 can be used to store computer executable program codes, which include instructions. The internal memory may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, at least one application required for a function, etc. The data storage area may store data (such as input data, output data) created during the use of the electronic device, etc. In addition, the internal memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor executes various functional applications and data processing of the electronic device by running instructions stored in the internal memory and / or instructions stored in a memory provided in the processor.

[0105] It is to be understood that the structure illustrated in the embodiment of the present application is only an example and does not constitute a limitation on the electronic device. The electronic device in the embodiment of the present application may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0106] An embodiment of the present application also provides a computer-readable storage medium, which includes a stored program, wherein when the program is executed by a processor, the large model long text processing capability evaluation method as described in the above embodiment is implemented.

[0107] An embodiment of the present application also provides a computer program product, which includes a program. When the program is executed by an electronic device, the electronic device implements the large model long text processing capability evaluation method as described in the above embodiment.

[0108] An embodiment of the present application also provides a chip system, including: a communication interface for inputting and / or outputting data; a processor for executing a computer executable program, so that a device equipped with the chip system executes the large model long text processing capability evaluation method as described in the above embodiment.

[0109] The electronic device involved in this application may be any product such as a personal computer (PC), a server, etc.

[0110] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk), etc.

[0111] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0112] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for evaluating the processing capability of large models and long texts, characterized in that: include: Obtaining a first evaluation task, the first evaluation task including a first task example and a first task question, the first task example including a first example content and a first example result corresponding to the first example content, the first task question being a question for obtaining a corresponding result for a second example content, the second example content being different from the first example content; Get the distraction text; Inserting the first task example into the interference text, and combining the interference text into which the first task example is inserted and the first task question to obtain a task text; Executing a first test, the first test comprising: inputting the task text into a large model to obtain a first answer output by the large model based on the first task question; An evaluation result of the large model is obtained according to the first answer.

2. The method according to claim 1, characterized in that Before obtaining the first evaluation task, the method further includes: Get multiple test samples; Replacing the semantic tags in the multiple test samples with preset tags; At least part of the multiple test examples are the first task examples, at least one of the multiple test examples is the first task problem, and the first task problem is different from the first task example.

3. The method according to claim 2, characterized in that Before obtaining the first evaluation task, the method further includes: Performing a pre-test, wherein the pre-test includes a plurality of second tests based on an i-th number of second task samples, where i is 1, 2, ..., n, and n is an integer greater than 1; Each time the second test comprises: inputting the i-th number of second task examples and at least one second task question into the large model, and obtaining a second answer output by the large model based on the at least one second task question; The i-th number of second task examples and the at least one second task problem are part of the multiple test examples, and the i-th number of second task examples are different in any different second tests under the same i value; Obtaining the accuracy of the second answers of multiple second tests of the i-th number of second task examples; The i value corresponding to the highest accuracy of the second answer in multiple second tests of the i-th number of second task examples is a, and the a-th number is the number of the first task examples in the first evaluation task.

4. The method according to claim 3, characterized in that The performing of the first test comprises performing the first test a plurality of times; The method of obtaining the evaluation result of the large model based on the first answer includes: calculating the depreciation rate, which is equal to (A2-A1) / A2, wherein A2 is the accuracy of the second answer in multiple second tests of the a-th number of second task samples, and A1 is the accuracy of the first answer in multiple first tests.

5. The method according to claim 4, characterized in that In any different times of the first test, at least one of the content of the interference text, the position where the first task sample is inserted into the interference text, and the content of the first assessment task is different.

6. The method according to any one of claims 2 to 5, characterized in that The test examples are context-based learning (ICL) task test examples.

7. The method according to claim 1, characterized in that The first evaluation task is a large-scale multi-task language understanding MMLU task.

8. The method according to claim 7, characterized in that Before obtaining the first evaluation task, the method further includes: Performing multiple third tests, each of which includes: inputting a third task example and a third task question into the large model, and obtaining a third answer output by the large model based on the third task question; The third task example and the third task problem are examples and problems corresponding to the MMLU task; In any different times of the third test, the third task samples are different; Obtaining a third test accuracy rate, where the third test accuracy rate is the accuracy rate of the third answers of the multiple third tests; The performing of the first test comprises performing the first test a plurality of times; In any different times of the first test, at least one of the content of the interference text, the position where the first task sample is inserted into the interference text, and the content of the first assessment task is different; The obtaining of the evaluation result of the large model according to the first answer includes: calculating a loss rate, wherein the loss rate is equal to (A3-A1) / A3, wherein A3 is the accuracy of the third test, and A1 is the accuracy of the first answer of multiple first tests.

9. An electronic device, characterized in that: The electronic device comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 8.

Images