Large language model evaluation method, system and device, medium and program product

Abstract

The invention discloses a large language model evaluation method, system and device, a medium and a program product, and the method comprises the steps: firstly, carrying out the pollution detection of an obtained benchmark test data set through a data detection method, obtaining first suspected pollution data, and then according to the type of the benchmark test data set, obtaining second suspected pollution data; data rewriting is conducted on the first suspected pollution data to obtain second suspected pollution data, data detection is conducted on the second suspected pollution data again through a data detection method till a preset stop condition is met, a benchmark test data set after pollution removal is obtained, and finally the data of the second suspected pollution data are detected on the basis of the benchmark test data set after pollution removal. And evaluating the large language model. Potential pollution of the benchmark test data set can be reduced, performance expansion of the large language model caused by remembering the leaked benchmark test is effectively relieved, the evaluation result of the large language model is more credible, and the real ability of the model can be better reflected.

Description

Technical Field

[0001] The present invention relates to the technical field of large language model evaluation, and in particular to a large language model evaluation method, system, device, medium and program product. Background Art

[0002] Large Language Models (LLMs) are artificial intelligence models designed to understand and generate human language by training on massive text datasets, and have demonstrated their potential in a wide range of applications. Therefore, evaluating the capabilities of large language models is crucial because it not only helps to rank models, but also helps to distinguish valuable work and effective model improvement strategies.

[0003] The large language model benchmark is a set of carefully designed test tasks, questions, and datasets that follow a standardized process to evaluate the performance of language models. However, current large language models are performing better and better in various benchmarks, but their performance in real applications does not always match the benchmark results. This is because the data used by large language models in the benchmarks is intentionally or unintentionally contaminated. Therefore, the impact of potential data contamination on the evaluation of large language models has prompted researchers to establish new benchmarks to more accurately evaluate model performance, but creating new benchmarks is time-consuming and expensive, and no matter what kind of benchmark, once it is made public, there is a risk of being leaked. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a large language model evaluation method, system, device, medium and program product, which can reduce the pollution degree of the benchmark test data set, thereby improving the evaluation accuracy of the large language model.

[0005] To solve the above technical problems, an embodiment of the present invention provides a large language model evaluation method, including:

[0006] Get the benchmark dataset;

[0007] Performing pollution detection on the benchmark test data set by a data detection method to obtain first suspected pollution data;

[0008] rewriting the first suspected contaminated data according to the type of the benchmark test data set to obtain second suspected contaminated data; wherein the data rewriting does not change the difficulty of the first suspected contaminated data;

[0009] Re-detecting the second suspected contaminated data by a data detection method until a preset stop condition is met, thereby obtaining a decontaminated benchmark test data set;

[0010] The large language model is evaluated based on the decontaminated benchmark dataset.

[0011] Preferably, the benchmark test data set includes at least one of a mathematical reasoning benchmark test data set and a knowledge and language comprehension benchmark test data set.

[0012] As an improvement of the above solution, the step of rewriting the first suspected polluted data according to the type of the benchmark test data set to obtain the second suspected polluted data includes:

[0013] If the benchmark test data set is a mathematical reasoning benchmark test data set, redesigning the problem scenario for the first suspected contaminated data based on the calculation logic and answer structure of the original mathematical reasoning problem to obtain second suspected contaminated data;

[0014] If the benchmark test data set is a knowledge and language comprehension benchmark test data set, then the original knowledge and language comprehension questions and options of the first suspected contaminated data are synonymously rewritten without changing the core proper nouns and any numbers to obtain the second suspected contaminated data.

[0015] As an improvement of the above solution, the method of performing pollution detection on the benchmark test data set by using a data detection method to obtain first suspected pollution data includes:

[0016] Calculating the average probability of uncommon words in the text of the benchmark test dataset;

[0017] The contamination status of the benchmark test data set is determined according to the average probability to obtain first suspected contaminated data.

[0018] To solve the above technical problems, an embodiment of the present invention further provides a large language model evaluation system, including:

[0019] A benchmark test data set acquisition module, used to acquire a benchmark test data set;

[0020] A first data pollution detection module, configured to perform pollution detection on the benchmark test data set by using a data detection method to obtain first suspected pollution data;

[0021] A data rewriting module, configured to rewrite the first suspected polluted data according to the type of the benchmark test data set to obtain second suspected polluted data; wherein the data rewriting does not change the difficulty of the suspected polluted data;

[0022] A second data contamination detection module, configured to re-detect the second suspected contaminated data using a data detection method until a preset stop condition is met, thereby obtaining a decontaminated benchmark test data set;

[0023] A model evaluation module is used to evaluate the large language model based on the decontaminated benchmark test dataset.

[0024] As an improvement of the above solution, the data rewriting module is specifically used for:

[0025] If the benchmark test data set is a mathematical reasoning benchmark test data set, redesigning the problem scenario for the first suspected contaminated data based on the calculation logic and answer structure of the original mathematical reasoning problem to obtain second suspected contaminated data;

[0026] If the benchmark test data set is a knowledge and language comprehension benchmark test data set, then the original knowledge and language comprehension questions and options of the first suspected contaminated data are synonymously rewritten without changing the core proper nouns and any numbers to obtain the second suspected contaminated data.

[0027] As an improvement of the above solution, the first data pollution detection module is specifically used to:

[0028] Calculating the average probability of uncommon words in the text of the benchmark test dataset;

[0029] The contamination status of the benchmark test data set is determined according to the average probability to obtain first suspected contaminated data.

[0030] To solve the above technical problems, an embodiment of the present invention further provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the large language model evaluation method as described in any one of the above when executing the computer program.

[0031] To solve the above technical problems, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the large language model evaluation methods described above.

[0032] To solve the above technical problems, an embodiment of the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the large language model evaluation method as described in any one of the above.

[0033] Compared with the prior art, the embodiment of the present invention provides a large language model evaluation method, system, device, medium and program product. The method first performs pollution detection on the acquired benchmark test data set through a data detection method to obtain first suspected polluted data, and then rewrites the first suspected polluted data according to the type of the benchmark test data set to obtain second suspected polluted data, and re-performs data detection on the second suspected polluted data through a data detection method until a preset stop condition is met to obtain a decontaminated benchmark test data set, and finally evaluates the large language model based on the decontaminated benchmark test data set. The present invention performs pollution detection and rewrites the benchmark test data set to remove the highly polluted part of the benchmark test data set, reduces the potential pollution of the benchmark test, effectively reduces the performance expansion of the large language model caused by remembering the leaked benchmark test, and makes the evaluation result of the large language model more credible and better reflects the real ability of the model; at the same time, the present invention avoids the high overhead problem of reconstructing the data set, so that the tester does not need to create a new benchmark test, and can evaluate the large language model through the existing benchmark test data set, thereby improving the efficiency of the large language model evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solution of the present invention, the drawings used in the implementation mode will be briefly introduced below. Obviously, the drawings described below are only some implementation modes of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 is a flow chart of a large language model evaluation method provided by an embodiment of the present invention;

[0036] Figure 2 It is a structural block diagram of a large language model evaluation system provided by an embodiment of the present invention;

[0037] Figure 3 It is a structural block diagram of a terminal device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] It should be noted that since many current benchmarks are completely public, data leakage varies, which affects the authenticity and fairness of large language model evaluation. And as large language models are trained on larger and larger training data sets, the possibility of benchmark-related data contaminating the training data set will also increase, causing LLMs to inadvertently "cheat". Therefore, the negative impact of data contamination needs to be avoided or mitigated.

[0040] Typically, data decontamination is applied during the model training phase, and model developers use various methods to remove overlaps between pre-training data and evaluation data. Traditional contamination detection methods for large language models generally directly calculate the overlap between pre-training data and evaluation data sets, including indexing and matching using n-gram analysis (a data mining method based on text sequences) and BM25 (Best Matching 25, an algorithm for information retrieval and text mining). However, as pre-training data grows exponentially, even simple n-gram statistics become extremely resource-consuming. Therefore, the present invention believes that data contamination detection is still a key issue and should be addressed in benchmarking.

[0041] In addition, new datasets can be created to avoid contamination. For example, some researchers have avoided model contamination by strictly following the chronological order and using the text published in the most recent time window to build a new question-answer set from the latest Wikipedia data. However, this method is also temporary; once a new dataset is released, it is exposed to the risk of leakage, needs to be constantly updated, and makes old benchmark results obsolete. Similarly, some researchers have created a new dataset GSM1K to ensure comparability in key indicators such as human solution rate, number of solution steps, and answer value. They prevent data leakage by not publishing the dataset. However, creating a new benchmark is time-consuming and expensive, and subsequent maintenance will also be a large expense. Therefore, the present invention believes that this is not an ideal solution to test data contamination.

[0042] See also Figure 1 , is a flow chart of a large language model evaluation method provided by an embodiment of the present invention, such as Figure 1 As shown, the large language model evaluation method includes steps S1 to S5:

[0043] S1. Obtain benchmark dataset;

[0044] Specifically, the benchmark test data set includes at least one of a mathematical reasoning benchmark test data set and a knowledge and language comprehension benchmark test data set.

[0045] It is understandable that in the specific implementation, we must first determine the specific task of the model to be evaluated, such as mathematical reasoning ability or knowledge and language understanding processing, and then select a benchmark dataset for it. Specifically, for the determined task, select a widely used and recognized benchmark dataset. These datasets are usually maintained by research institutions or organizations and provide training sets and test sets.

[0046] For example, the knowledge and language processing task selects MMLU (Massive Multitask Language Understanding), which aims to evaluate the language understanding ability of large models. It is one of the most famous large-model semantic understanding evaluations. This evaluation benchmark combines test sets of 57 subjects such as mathematics, physics, history, law, medicine, and ethics, which can comprehensively evaluate large language models.

[0047] The math reasoning task selected is GSM8K (Grade School Math 8K), which contains 8,500 high-quality elementary school math problem datasets in diverse languages. It is one of the benchmarks for evaluating the math reasoning ability of large language models. GSM8K is larger in scale, more diverse in language, and more challenging in questions than other math word problem datasets, which helps evaluate the ability of large language models to solve basic math problems and reasoning.

[0048] S2. Performing pollution detection on the benchmark test data set by a data detection method to obtain first suspected pollution data;

[0049] It should be noted that, before the evaluation, the embodiment of the present invention performs a contamination detection on each sample in the benchmark test data set for the evaluated model, so that the original benchmark test data set can be divided into two parts: non-contaminated and suspected contaminated. Therefore, when the benchmark test data set is subjected to contamination detection by the data detection method, non-contaminated data is also obtained.

[0050] In a preferred embodiment, performing pollution detection on the benchmark test data set by a data detection method to obtain first suspected pollution data includes:

[0051] Calculating the average probability of uncommon words in the text of the benchmark test dataset;

[0052] The contamination status of the benchmark test data set is determined according to the average probability theory to obtain first suspected contaminated data.

[0053] Exemplarily, the data detection method adopted in the embodiment of the present invention is MIN-K% PROB (minimum K% probability), and the goal of the data detection method is to determine whether the text X appears in the pre-training data of the large language model.

[0054] If the probability P(x i ) of obtaining each word segment x in text X is obtained, and the K word segment probabilities with the lowest probabilities are selected i . Then calculate the average value of these probabilities:

[0055]

[0056] Among them, is the probability of the j-th word segment with the lowest probability, and a threshold ε is set to determine whether X appears in the pre-training data.

[0057] It should be noted that the Min-K%Prob method is simple and efficient. It does not need to know the pre-training corpus of the large language model. By calculating the probabilities of uncommon words in the text, it can judge whether the text belongs to the pre-training data of the model. This method focuses on the probabilities of uncommon words in the text. Common words in the large language model (such as "and", "is", "of") will have relatively high probabilities, while uncommon words such as technical terms or rare names will have relatively low probabilities. Therefore, to judge whether the model has "learned" a certain text, mainly look at the probabilities of uncommon words in the text. If the probabilities of these words in the model are relatively high, it may mean that the model has learned this text; on the contrary, relatively low probabilities indicate that the text may not be learned by the model.

[0058] It is worth noting that for the data detection in the implementation of the present invention, there are no other restrictions except for the need to perform contamination detection on the fine-grained level of the sample. Therefore, the embodiments of the present invention do not depend on a specific detection method. Using any data detection method, or a more accurate detection method in the future, the large language model evaluation method provided by the embodiments of the present invention is compatible.

[0059] S3. According to the type of the benchmark test data set, rewrite the first suspected contaminated data to obtain second suspected contaminated data; wherein, the data rewrite does not change the difficulty of the first suspected contaminated data;

[0060] Specifically, in the embodiments of the present invention, the suspected contaminated data divided by data detection is rewritten without changing the difficulty.

[0061] In a preferred embodiment, the rewriting the first suspected contaminated data according to the type of the benchmark test data set to obtain second suspected contaminated data includes:

[0062] If the benchmark test data set is a mathematical reasoning benchmark test data set, then based on the calculation logic and answer structure of the original mathematical reasoning problem, redesign the problem scenario for the first suspected contaminated data to obtain second suspected contaminated data;​​

[0063] If the benchmark test data set is a knowledge and language comprehension benchmark test data set, then the original knowledge and language comprehension questions and options of the first suspected contaminated data are synonymously rewritten without changing the core proper nouns and any numbers to obtain second suspected contaminated data.

[0064] It should be noted that the embodiments of the present invention define how to rewrite mathematical reasoning ability questions and language comprehension processing questions respectively so as not to change the core content and difficulty of the questions.

[0065] For mathematical reasoning ability questions, such as GSM8K, the embodiment of the present invention redesigns the problem context based on the calculation logic and answer structure of the original mathematical problem, which can ensure that the problem becomes more diverse while maintaining the data difficulty and consistency of the answer. For example, the application context in the original problem, such as eggs, can be changed to candy, and the characters involved can also be replaced with different characters, and then all the numbers involved and the mathematical steps in the answer are kept consistent, thus ensuring the coherence of the content being evaluated and the consistency of the difficulty.

[0066] Exemplarily, the embodiment of the present invention provides an effect of rewriting a mathematical problem in a GSM8K data set as follows:

[0067] 1. Data suspected of contamination:

[0068] Question: Xiao Ming decided to run sprints three times a week, 60 meters each time. How many meters does he run in total per week?

[0069] Calculation steps: He sprinted 3*3=9 times, so he ran 9*60=540 meters.

[0070] Answer: He ran a total of 540 meters every week.

[0071] 2. Rewrite:

[0072] Question: Xiaohua plans to swim 3 laps per week, each lap is 60 meters. What is the total distance he swims in a week?

[0073] Calculation steps: She swims 3*3=9 laps a week, so she swims 9*60=540 meters.

[0074] Answer: The total distance she swims in one lap is 360 meters.

[0075] For knowledge and language understanding processing problems, such as MMLU, the inventors found that rewriting these problems requires a huge knowledge base, and any changes in the background need to verify their correctness. Therefore, the embodiment of the present invention chooses to rewrite the questions and options of this knowledge and language processing synonymously, but does not change the core proper nouns and any numbers. For example, the question "Discuss the impact of climate change on the Arctic in recent years" is rewritten as "Explore the changes in the Arctic caused by climate change in recent years". In addition, it is also possible to avoid using uncommon words that may increase the difficulty of understanding, thereby ensuring the consistency of knowledge points and difficulty.

[0076] It is worth noting that the rewriting method for knowledge and language understanding processing problems is synonymous rewriting, which is very universal and can cover almost other problems. Therefore, the embodiment of the present invention can also accommodate better rewriting methods in the future. Of course, in the specific implementation, designing a rewriting scheme for the more detailed tasks can also further improve the effect.

[0077] S4, re-detecting the second suspected contaminated data by a data detection method until a preset stop condition is met, thereby obtaining a decontaminated benchmark test data set;

[0078] It is worth noting that because some large language models may have undergone a large amount of same-domain training, the result of a rewrite may still be in the model memory. Therefore, the embodiment of the present invention further re-tests the rewritten data, and loops the data detection and rewriting in the previous steps until the rewritten content passes the test or the maximum number of rewrites is reached.

[0079] S5. Evaluate the large language model based on the decontaminated benchmark test dataset.

[0080] Exemplarily, taking the evaluation of a large language model named A as an example, the decontaminated benchmark test dataset is first divided into a training set, a validation set, and a test set; then the training set is used to fine-tune the large language model A, the fine-tuned large language model A is evaluated on the validation set, and the summary quality index of the large language model A on the validation set is calculated; the fine-tuned large language model A is evaluated on the test set, and the summary quality index of the large language model A on the test set is calculated; finally, the performance of the fine-tuned large language model A on the validation set and the test set is analyzed, and compared with the basic large language model A.

[0081] In summary, the embodiment of the present invention provides a large language model evaluation method, which first performs pollution detection on the acquired benchmark test data set through a data detection method to obtain first suspected polluted data, and then rewrites the first suspected polluted data according to the type of the benchmark test data set to obtain second suspected polluted data, and re-performs data detection on the second suspected polluted data through a data detection method until a preset stop condition is met to obtain a decontaminated benchmark test data set, and finally evaluates the large language model based on the decontaminated benchmark test data set. The present invention performs pollution detection and rewrites the benchmark test data set, removes the highly polluted part of the benchmark test data set, reduces the potential pollution of the benchmark test, effectively reduces the performance expansion of the large language model caused by remembering the leaked benchmark test, and makes the evaluation result of the large language model more credible and better reflects the real ability of the model; at the same time, the present invention avoids the high overhead problem of reconstructing the data set, so that the tester does not need to create a new benchmark test, and can evaluate the large language model through the existing benchmark test data set, thereby improving the efficiency of the large language model evaluation.

[0082] On the basis of the above-mentioned method items, the present invention provides corresponding embodiments of the system items.

[0083] See also Figure 2 , Figure 2 : is a structural block diagram of a large language model evaluation system provided by an embodiment of the present invention. The large language model evaluation system comprises:

[0084] A benchmark test data set acquisition module 21, used to acquire a benchmark test data set;

[0085] A first data pollution detection module 22, configured to perform pollution detection on the benchmark test data set by using a data detection method to obtain first suspected pollution data;

[0086] A data rewriting module 23, configured to rewrite the first suspected polluted data according to the type of the benchmark test data set to obtain second suspected polluted data; wherein the data rewriting does not change the difficulty of the suspected polluted data;

[0087] A second data contamination detection module 24, configured to re-detect the second suspected contaminated data using a data detection method until a preset stop condition is met, thereby obtaining a decontaminated benchmark test data set;

[0088] The model evaluation module 25 is used to evaluate the large language model based on the decontaminated benchmark test data set.

[0089] In a preferred embodiment, the data rewriting module 23 is specifically used for:

[0090] If the benchmark test data set is a mathematical reasoning benchmark test data set, redesigning the problem scenario for the first suspected contaminated data based on the calculation logic and answer structure of the original mathematical reasoning problem to obtain second suspected contaminated data;

[0091] If the benchmark test data set is a knowledge and language comprehension benchmark test data set, then the original knowledge and language comprehension questions and options of the first suspected contaminated data are synonymously rewritten without changing the core proper nouns and any numbers to obtain the second suspected contaminated data.

[0092] In a preferred embodiment, the first data pollution detection module 22 is specifically used to:

[0093] Calculating the average probability of uncommon words in the text of the benchmark test dataset;

[0094] The contamination status of the benchmark test data set is determined according to the average probability to obtain first suspected contaminated data.

[0095] It should be noted that a large language model evaluation system provided in an embodiment of the present invention is used to execute all process steps of a large language model evaluation method in the above embodiment. The working principles and beneficial effects of the two correspond one to one, and thus will not be described in detail.

[0096] The embodiment of the present invention also provides a terminal device, such as Figure 3 , which is a structural block diagram of a preferred embodiment of a terminal device provided by the present invention. The terminal device includes a processor 31, a memory 32, and a computer program stored in the memory 32 and configured to be executed by the processor 31. When the processor 31 executes the computer program, the large language model evaluation method described in any of the above embodiments is implemented.

[0097] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the large language model evaluation method described in any of the above embodiments.

[0098] When the processor 31 executes the computer program, the steps in the above large language model evaluation method embodiment are implemented, for example: Figure 1 Alternatively, when the processor 31 executes the computer program, the functions of each module in the above-mentioned large language model evaluation system embodiment are realized, for example Figure 3 The functions of each module of the large language model evaluation system are shown.

[0099] Preferably, the computer program can be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor 31 to implement the present invention. The one or more modules / units can be a series of computer program instruction segments that can implement specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0100] The processor 31 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor, or the processor 31 can be any conventional processor. The processor 31 is the control center of the terminal device, and various parts of the terminal device are connected using various interfaces and lines.

[0101] The memory 32 mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory 32 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card), etc., or the memory 32 can also be other volatile solid-state storage devices.

[0102] It should be noted that the above terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 3 The structural block diagram shown is only a structural example of the above-mentioned terminal device and does not constitute a structural limitation of the above-mentioned terminal device. The above-mentioned terminal device may include more or fewer components than shown in the figure, or combine certain components, or different components.

[0103] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A large language model evaluation method, characterized in that: include: Get the benchmark dataset; Performing pollution detection on the benchmark test data set by a data detection method to obtain first suspected pollution data; rewriting the first suspected contaminated data according to the type of the benchmark test data set to obtain second suspected contaminated data; wherein the data rewriting does not change the difficulty of the first suspected contaminated data; Re-detecting the second suspected contaminated data by a data detection method until a preset stop condition is met, thereby obtaining a decontaminated benchmark test data set; The large language model is evaluated based on the decontaminated benchmark dataset.

2. The large language model evaluation method according to claim 1, characterized in that: The benchmark test data set includes at least one of a mathematical reasoning benchmark test data set and a knowledge and language comprehension benchmark test data set.

3. The large language model evaluation method according to claim 1, characterized in that: The step of rewriting the first suspected polluted data according to the type of the benchmark test data set to obtain second suspected polluted data includes: If the benchmark test data set is a mathematical reasoning benchmark test data set, redesigning the problem scenario for the first suspected contaminated data based on the calculation logic and answer structure of the original mathematical reasoning problem to obtain second suspected contaminated data; If the benchmark test data set is a knowledge and language comprehension benchmark test data set, then the original knowledge and language comprehension questions and options of the first suspected contaminated data are synonymously rewritten without changing the core proper nouns and any numbers to obtain the second suspected contaminated data.

4. The large language model evaluation method according to claim 1, characterized in that: The step of performing pollution detection on the benchmark test data set by using a data detection method to obtain first suspected pollution data includes: Calculating the average probability of uncommon words in the text of the benchmark test dataset; The contamination status of the benchmark test data set is determined according to the average probability to obtain first suspected contaminated data.

5. A large language model evaluation system, characterized in that: include: A benchmark test data set acquisition module, used to acquire a benchmark test data set; A first data pollution detection module, configured to perform pollution detection on the benchmark test data set by using a data detection method to obtain first suspected pollution data; A data rewriting module, configured to rewrite the first suspected polluted data according to the type of the benchmark test data set to obtain second suspected polluted data; wherein the data rewriting does not change the difficulty of the suspected polluted data; A second data contamination detection module, configured to re-detect the second suspected contaminated data using a data detection method until a preset stop condition is met, thereby obtaining a decontaminated benchmark test data set; A model evaluation module is used to evaluate the large language model based on the decontaminated benchmark test dataset.

6. The large language model evaluation system according to claim 5, characterized in that: The data rewriting module is specifically used for: If the benchmark test data set is a mathematical reasoning benchmark test data set, redesigning the problem scenario for the first suspected contaminated data based on the calculation logic and answer structure of the original mathematical reasoning problem to obtain second suspected contaminated data; If the benchmark test data set is a knowledge and language comprehension benchmark test data set, then the original knowledge and language comprehension questions and options of the first suspected contaminated data are synonymously rewritten without changing the core proper nouns and any numbers to obtain the second suspected contaminated data.

7. The large language model evaluation system according to claim 5, characterized in that: The first data pollution detection module is specifically used to: Calculating the average probability of uncommon words in the text of the benchmark test dataset; The contamination status of the benchmark test data set is determined according to the average probability to obtain first suspected contaminated data.

8. A terminal device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the large language model evaluation method according to any one of claims 1 to 4 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the large language model evaluation method according to any one of claims 1 to 4.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the large language model evaluation method as claimed in any one of claims 1 to 4 is implemented.

Images