Large model capability assessment method and device, and storage medium

Abstract

The invention discloses a large model capability assessment method and device and a storage medium, and relates to the technical field of artificial intelligence and deep learning, and the method comprises the steps: carrying out the reasoning operation of test data through a preset reasoning model, and obtaining model answer data, the test data does not contain a data set with label information; a target scoring mode matched with the data size of the model answer data is determined, the model answer data is scored according to the target scoring mode, an evaluation score is obtained, and the target scoring mode scores the model answer data through at least two ability dimensions; the evaluation score is output and forms score data with the test data. According to the invention, the accuracy of large model capability evaluation in the government affair circulation field can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and deep learning technology, and in particular to a large model capability assessment method, device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models have shown great potential in various fields. However, existing evaluation methods have some shortcomings in measuring the capabilities of these models. Although traditional manual evaluation methods can provide in-depth semantic understanding, they are costly and susceptible to subjectivity. Automated evaluation indicators, such as BLEU (Bilingual Evaluation Understudy) and ROUGE (Recall-Oriented Understudy for Gisting Evaluation), can quickly and consistently evaluate models, but often fail to fully capture the performance of models in complex tasks, especially in logical reasoning and user demand satisfaction in the field of government affairs circulation. In addition, although the method of using more powerful models as referees can provide a higher level of evaluation, this method may have problems with model bias and mutual influence, resulting in inaccurate evaluation results.

[0003] Therefore, how to improve the accuracy of large-scale model capability assessment in the field of government affairs circulation is a problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of this application is to provide a large model capability assessment method, device and storage medium, aiming to solve the technical problem of how to improve the accuracy of large model capability assessment in the field of government affairs circulation.

[0005] To achieve the above objectives, the present application provides a large model capability evaluation method, the large model capability evaluation method comprising:

[0006] Performing inference operations on the test data using a preset inference model to obtain model answer data, wherein the test data does not include a data set with label information;

[0007] Determining a target scoring method that matches the data volume of the model answer data, and scoring the model answer data according to the target scoring method to obtain an evaluation score, wherein the target scoring method scores the model answer data according to at least two capability dimensions;

[0008] The evaluation score is output and forms score data together with the test data.

[0009] In one embodiment, the target scoring method includes a reference answer and a scoring rule, and the step of scoring the model answer data according to the target scoring method to obtain an evaluation score includes:

[0010] Compare the model answer data with the reference answer, obtain the shortcomings of the model answer data and output an explanation of the shortcomings;

[0011] Based on the weakness explanation, scoring the model answer data by at least two capability dimensions to obtain a sub-score corresponding to each capability dimension;

[0012] According to the scoring rules, the sub-scores are comprehensively scored to obtain an evaluation score.

[0013] In one embodiment, the capability dimension includes at least factual correctness and user needs, and the step of comprehensively scoring the sub-scores according to the scoring rule to obtain an evaluation score further includes:

[0014] If the sub-score of factual correctness or user needs is lower than or equal to the preset minimum score, an evaluation score is obtained based on the sub-scores of factual correctness and user needs, and the evaluation score is lower than or equal to the preset minimum score;

[0015] If the sub-scores of factual correctness and user needs are higher than the preset minimum score, the sub-scores are averaged to obtain an evaluation score.

[0016] In one embodiment, the target scoring method includes a reference answer and a score calculation function, and the step of scoring the model answer data according to the target scoring method to obtain an evaluation score includes:

[0017] Input the reference answer and the model answer data into the score calculation function to calculate the sub-score corresponding to each ability dimension;

[0018] The sub-scores are averaged to obtain an assessment score.

[0019] In one embodiment, the step of determining a target scoring method that matches the data volume of the model answer data includes:

[0020] Comparing the data volume of the model answer data with the preset data volume to obtain a comparison result;

[0021] A scoring method that matches the comparison result is determined from among a plurality of preset scoring methods as a target scoring method that matches the data volume of the model answer data.

[0022] In one embodiment, before the step of performing an inference operation on the test data by using a preset inference model to obtain model answer data, the step includes:

[0023] Get the original data set;

[0024] Determine whether the original data set is an open source evaluation set, where the open source evaluation set is a data set that has been publicly released and allowed to be freely downloaded and used;

[0025] If the original data set is not an open source evaluation set, performing data cleaning on the original data set to obtain cleaned data;

[0026] Inputting the cleaning data into a preset core module for processing to obtain test data;

[0027] If the original data set is an open source evaluation set, the original data set is input into the core module for processing to obtain test data.

[0028] In one embodiment, the step of inputting the cleaning data into a preset core module for processing to obtain test data includes:

[0029] generating key information data based on the cleansed data;

[0030] Generate question-answering questions based on the key information data;

[0031] Determining whether the question-and-answer question is available, and dividing the question-and-answer question into available questions and unavailable questions;

[0032] The available questions are collected to form test data.

[0033] In one embodiment, the test data includes a logic capability data set and a government affairs flow data set, and the model answer data includes first model answer data and second model answer data;

[0034] The step of performing an inference operation on the test data by using a preset inference model to obtain model answer data includes:

[0035] Performing an inference operation on the logic capability data set through a preset inference model to obtain first model answer data, wherein the first model answer data is used to score the general capability;

[0036] The government affairs circulation data set is subjected to inference operation through a preset inference model to obtain second model answer data, and the second model answer data is used to score the government affairs circulation capability.

[0037] In one embodiment, after the step of outputting the evaluation score and forming the score data together with the test data, the following steps are included:

[0038] The test data in the score data are stored in a preset corpus, and the preset corpus is fine-tuned based on the evaluation scores in the score data.

[0039] In addition, to achieve the above-mentioned purpose, the present application also provides a large model capability assessment device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program is configured to execute the steps of the large model capability scoring method as described above.

[0040] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium, and a program for implementing the large model capability evaluation method is stored on the computer-readable storage medium. The program for implementing the large model capability evaluation method is executed by a processor to implement the steps of the large model capability evaluation method as described above.

[0041] The present application provides a large model capability assessment method, which first obtains the model answer data generated by the preset thrust model by inputting the test data into the preset reasoning model that needs to be evaluated for capability, wherein the test data here is the test data that has been processed in a targeted manner in the field of government affairs circulation, and the test data does not contain a data set with label information, so that the test data is more suitable for the assessment of the field of government affairs circulation; then, the matching target scoring method is determined according to the data volume of the model answer data, and the model answer data is scored by the target scoring method to obtain the evaluation score. The model answer data with different data volumes are scored by different scoring methods to make the scoring more accurate; finally, the evaluation score is output, and the test data and the evaluation score are integrated to obtain the score data.

[0042] In summary, this application uses test data that has been specifically processed in the field of government affairs circulation, and selects the scoring method based on the amount of data answered by the model. It overcomes the technical defect that the single scoring method cannot accurately evaluate the capabilities of the large model, and improves the accuracy of the large model capability assessment in the field of government affairs circulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0045] Figure 1 A flowchart diagram of the first embodiment of the large model capability assessment method of the present application;

[0046] Figure 2 A flowchart diagram of the second embodiment of the large model capability assessment method of the present application;

[0047] Figure 3 A schematic diagram of a simplified process of a large model capability evaluation method provided in one embodiment of the present application;

[0048] Figure 4 This is a schematic diagram of the module structure of a large model capability evaluation device according to an embodiment of the present application;

[0049] Figure 5 It is a schematic diagram of the structure of a large model capability evaluation device in an embodiment of the present application.

[0050] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0052] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0053] The main solution of the present application is: to perform inference operations on test data through a preset inference model to obtain model answer data, wherein the test data does not contain a data set with label information; to determine a target scoring method that matches the data volume of the model answer data, and to score the model answer data according to the target scoring method to obtain an evaluation score, wherein the target scoring method scores the model answer data through at least two capability dimensions; and to output the evaluation score and form score data together with the test data.

[0054] At present, when evaluating the capabilities of large models, three schemes are usually conceived and implemented: manual evaluation, evaluation through specific evaluation indicators, and evaluation through more powerful models as referees. Among them, manual evaluation usually involves professional evaluators reading the text generated by the model and scoring it according to a series of criteria. The disadvantage is that manual evaluation is costly and time-consuming, and may be affected by the subjectivity of the evaluators. Evaluation through specific evaluation indicators usually uses an automated evaluation system to evaluate the performance of the model according to predefined indicators, such as BLEU, ROUGE, etc. The disadvantage is that these indicators may not fully reflect the complex capabilities of the model and may sometimes ignore the semantic quality of the text. Evaluation through more powerful models as referees uses one or more more advanced language models to evaluate the performance of other models. The disadvantage is that the referee model itself may also have biases and limitations, and this method may introduce mutual influences between evaluation models. When evaluating large models in the field of government affairs circulation through the above schemes, the shortcomings of the schemes may lead to problems such as insufficient evaluation accuracy or high time costs.

[0055] Therefore, how to improve the accuracy of large-scale model capability assessment in the field of government affairs circulation is a problem that needs to be solved urgently.

[0056] This application uses test data that has been specifically processed in the field of government affairs circulation, and selects the scoring method based on the amount of data answered by the model. It overcomes the technical defect that the single scoring method cannot accurately evaluate the capabilities of the large model. It has unique advantages in terms of targeting, comprehensiveness, and degree of automation. It is particularly suitable for in-depth evaluation of the circulation classification capabilities of the large model in the professional field of government affairs, and improves the accuracy of the large model capability assessment in the field of government affairs circulation.

[0057] It should be noted that the executor of this embodiment may be a data processing system, which may include an original data set unit, an open source evaluation set judgment unit, a data cleaning unit, a core module unit, a test data generation unit, a model reasoning unit, an evaluation score calculation unit, a score data output unit and a system control unit. The original data set unit is used to store the original data to be processed; the open source evaluation set judgment unit is used to judge whether the original data set is an open source evaluation set; the data cleaning unit performs a cleaning operation on the original data set when the open source evaluation set judgment unit judges that the original data set is not an open source evaluation set; the core module unit receives the cleaned data and processes it; the test data generation unit generates test data after receiving the processed data from the core module unit; the model reasoning unit performs reasoning operations on the test data to obtain model answer data; the evaluation score calculation unit compares the model answer data with the reference answer data to calculate the evaluation score; the score data output unit outputs the evaluation result and forms score data together with the test data; the system control unit controls the operating status of the entire system. It may also be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a large model capability evaluation device capable of realizing the above functions, etc., which is not specifically limited in this embodiment. The following takes the data processing system as the execution subject as an example to illustrate this embodiment and the following embodiments.

[0058] Based on this, this application proposes a large model capability evaluation method of the first embodiment, please refer to Figure 1 The large model capability evaluation method includes steps S10 to S30:

[0059] Step S10, performing reasoning operations on the test data using a preset reasoning model to obtain model answer data, wherein the test data does not contain a data set with label information;

[0060] It should be noted that the preset reasoning model refers to the model that needs to be evaluated for the large model capability, which is the evaluation object of this embodiment. Test data refers to the test set used to evaluate the capability of the preset reasoning model. Model answer data refers to the answer output by the preset reasoning model based on the test data.

[0061] It is understandable that when the preset reasoning model analyzes and answers test data, the quality of the model's answer data involves two aspects: one is the preset reasoning model's ability to understand language logic, and the other is its ability to understand professional domain knowledge.

[0062] In a feasible embodiment, the test data includes a logic capability data set and a government affairs flow data set, the model answer data includes first model answer data and second model answer data, and step S10 includes steps S101 to S102:

[0063] Step S101, performing reasoning operations on a logic capability data set through a preset reasoning model to obtain first model answer data, where the first model answer data is used to score the general capability;

[0064] Step S102, performing reasoning operations on the government affairs flow data set through a preset reasoning model to obtain second model answer data, and the second model answer data is used to score the government affairs flow capability.

[0065] It should be noted that the logical ability data set refers to the test data for testing the language logic comprehension ability (general ability) of the preset reasoning model, and the government affairs circulation data set refers to the test data for testing the professional domain knowledge comprehension ability (government affairs circulation ability) of the preset reasoning model.

[0066] In addition, it should be noted that the first model answer data refers to the answer output by the preset reasoning model for the logical capability data set, and the second model answer data refers to the answer output by the preset reasoning model for the government affairs flow data set.

[0067] The large model capability evaluation method of this embodiment can focus on evaluating the performance of the preset reasoning model in government question answering, understanding and flow classification by dividing the test data into a logical capability data set and a government flow data set. This highly targeted test data helps to discover the advantages and disadvantages of the model in specific business scenarios, such as the government flow field, and provides a clear direction for improving the preset reasoning model.

[0068] Step S20, determining a target scoring method that matches the data volume of the model answer data, and scoring the model answer data according to the target scoring method to obtain an evaluation score, wherein the target scoring method scores the model answer data according to at least two capability dimensions;

[0069] It should be noted that the scoring method refers to the method used to measure and quantify the quality of the model's answer data, and the evaluation score refers to the score obtained by scoring the model's answer data according to the scoring method.

[0070] It is understandable that if the evaluation system is simplified and only focuses on a single capability dimension such as classification accuracy, this method cannot fully reflect the comprehensive capabilities of the model, resulting in insufficient evaluation accuracy.

[0071] In a feasible embodiment, the capability dimension includes seven dimensions: factual correctness, user needs, clarity, completeness, creativity, logical coherence, and fairness and accountability. Factual correctness refers to whether the information provided in the model answer data is accurate and based on reliable facts and data; user needs refer to whether the model answer data meets the purpose and needs of the user's questions and whether it responds to the user's questions in a comprehensive and appropriate manner; clarity refers to whether the model answer data is clearly expressed and easy to understand, and whether concise language and structure are used so that users can easily understand it; completeness refers to whether the model answer data provides sufficient information and details to meet the user's needs and whether important aspects are omitted; creativity refers to whether the model answer data is innovative or unique and whether it provides novel insights or solutions; logical coherence refers to whether the model answer data is consistent as a whole and whether it maintains logical coherence between different parts to avoid self-contradiction; fairness and accountability refers to whether the model answer data takes into account different views and positions, whether it provides fair information or suggestions, does not carry personal views or prejudices, does not favor one party, whether the suggestions or information provided are feasible, whether it bears certain responsibilities, and whether potential risks and consequences are considered. This embodiment comprehensively scores the model answer data through the above seven dimensions, which can improve the accuracy of the large model capability assessment method of this embodiment.

[0072] It is understandable that the capability dimension does not necessarily have to include the above seven dimensions, but it must at least include two dimensions: factual correctness and user needs. Other dimensions can be increased or decreased as appropriate based on actual conditions such as model computing power and user needs.

[0073] It should be noted that this embodiment has prepared multiple sets of scoring methods for different data amounts. In actual applications, the matching scoring method can be used as the target scoring method for scoring based on the data amount of the model answer data, so as to obtain a more targeted scoring.

[0074] In a feasible embodiment, the step of determining a target scoring method that matches the data volume of the model answer data in step S20 includes steps D10 to D20:

[0075] Step D10, comparing the data volume of the model answer data with the preset data volume to obtain a comparison result;

[0076] Step D20, determining a scoring method that matches the comparison result from among a plurality of preset scoring methods as a target scoring method that matches the data volume of the model answer data.

[0077] It is not difficult to understand that the preset data volume can be flexibly set by the user according to actual needs.

[0078] Exemplarily, the comparison result may be divided into: the data amount of the model answer data is greater than a preset data amount; and the data amount of the model answer data is less than or equal to the preset data amount.

[0079] In this embodiment, a set of scoring methods is prepared in advance for each comparison result, and the comparison result is associated with the scoring method through a mapping table or the like. Therefore, in actual applications, the data volume of the model answer data and the preset data volume are compared to obtain a comparison result, and the scoring method associated with the comparison result is queried to determine the scoring method that matches the comparison result from multiple scoring methods as the target scoring method that matches the data volume of the model answer data.

[0080] In this embodiment, by comparing the data volume of the model answer data with the preset data volume, a scoring method that matches the comparison result is selected from multiple pre-prepared scoring methods as a template scoring method that matches the data volume of the model answer data, that is, a scoring method for the data volume of the model answer data, thereby adopting a target scoring method that matches the data volume of the model answer data to score the model answer data, thereby obtaining a more targeted evaluation score.

[0081] In a feasible embodiment, the target scoring method includes reference answers and scoring rules. In step S20, the model answer data is scored according to the target scoring method, and the step of obtaining the evaluation score includes steps A10 to A30:

[0082] Step A10, comparing the model answer data with the reference answer, obtaining the shortcomings of the model answer data and outputting the explanation of the shortcomings;

[0083] In this embodiment, if the data volume of the model answer data is greater than the preset data volume, the scoring method including the reference answer and the scoring rules will be used as the target scoring method.

[0084] It should be noted that reference answers refer to answers prepared in advance based on questions in the test data, which can be used as part of the scoring method. Scoring rules refer to the scoring rules given based on the reference answers, which are similar to the evaluation criteria for college entrance examination essay marking. Explanation of shortcomings refers to the explanation of shortcomings after comparing the model answer data with the reference answers, including the causes of the shortcomings and the impact of the shortcomings.

[0085] It is understandable that before scoring, it is necessary to give the defects of the model's answer data compared with the reference answer, and explain the shortcomings. If the shortcomings and explanations of the shortcomings are not given, it may lead to scoring errors due to misunderstanding of the model's answer data. By outputting the shortcomings and explanations of the shortcomings, the accuracy of the subsequent scoring is guaranteed.

[0086] Step A20, based on the explanation of the shortcomings, scoring the model answer data by at least two capability dimensions to obtain a sub-score corresponding to each capability dimension;

[0087] It should be noted that the sub-score refers to the score given for a single ability dimension, representing the quality of the answer on a single ability dimension.

[0088] It is understandable that by scoring each capability dimension separately, the impact of each capability dimension on the quality of the model's answer data can be fully considered. For example, when the quality of the answer in the factual correctness dimension is low, the sub-score is 2 points. At this time, the model's answer data is inconsistent with the facts and is a low-quality answer. Even if the sub-scores of other capability dimensions such as logical coherence are high, it is still a low-quality answer. If the factual correctness dimension is not scored, the model's answer data cannot be accurately evaluated.

[0089] Step A30, comprehensively scoring the sub-scores according to the scoring rules to obtain an evaluation score.

[0090] It should be noted that the evaluation score refers to the overall score of the model's answer data, which is used to measure the overall answer quality of the model's answer data.

[0091] In a feasible embodiment, the scoring rules are input into the data processing system in the form of prompts to score the model answer data. The specific content of the prompt is:

[0092] "You are an assistant who is good at evaluating text quality. Please evaluate the quality of an AI assistant's answer to a user's question as an impartial judge. The type of answer you are evaluating is {category}. You need to evaluate the answer from the following dimensions: "factual correctness", "user needs", "clarity", "completeness", "logical coherence", "fairness and accountability" and "creativity".

[0093] We will provide you with user questions, high-quality reference answers, and answers from the AI ​​(Artificial Intelligence) assistant that you need to evaluate. When you start your evaluation, you need to follow the following process:

[0094] 1. Compare the AI ​​assistant’s answer with the reference answer, point out the deficiencies of the AI ​​assistant’s answer, and explain further.

[0095] 2. Evaluate the AI ​​assistant’s answers from different dimensions. After evaluating each dimension, give each dimension a score of 1 to 10.

[0096] 3. Finally, based on the evaluation of each dimension, give the AI ​​assistant’s answer an overall score of 1 to 10.

[0097] 4. Your scoring needs to be as strict as possible and follow the following scoring rules: In general, the higher the quality of the model's answer, the higher the score. Among them, the two dimensions of factual correctness and meeting user needs are the most important, and the scores of these two dimensions dominate the final comprehensive score. If one of the two requirements, factual correctness and meeting user needs, is scored 1 to 2 points, the comprehensive score of the AI ​​assistant's answer can only be 1 to 2 points.

[0098] When the model answer is irrelevant to the question, has fundamental factual errors, or generates harmful content, the total score must be 1 to 2 points;

[0099] When the model answer has no serious errors and is basically harmless, but the quality is low and does not meet user needs, the total score is 3 to 4 points;

[0100] When the model's answer basically meets the user's requirements, but performs poorly in some dimensions, the quality is medium and the total score can be 5 to 6 points;

[0101] When the model's answer quality is close to the reference answer and performs well in all dimensions, the total score is 7 to 8 points;

[0102] Only when the quality of the model's answer significantly exceeds the reference answer, fully solves the user's problems and all needs, and is close to full marks in all dimensions can it get 9 to 10 points.

[0103] As an example, the reference answer can get 8 points.

[0104] Remember that you must evaluate and explain before you score. After your explanation of each dimension, you need to add your score for that dimension. Then, at the end of your answer, return all your scores in the following dictionary format (including brackets), and make sure your scores are integers:

[0105] {{'Dimension 1': score, 'Dimension 2': score, ..., 'Comprehensive score': score}}, for example: {{'Factual accuracy': 9, 'User needs': 6, ..., 'Comprehensive score': 7}}.

[0106] User's question: {question};

[0107] [reference answer begins]{reference}[reference answer ends];

[0108] [assistant's answer begins]{answer}[assistant's answer ends];

[0109] Category options: [Simple understanding], [Knowledge application], [Logical reasoning], [Generative creation]."

[0110] It can be understood that, through the above prompt, this embodiment can score model answer data with a data volume greater than a preset data volume. Through automated scoring, the calculation steps of self-scoring are reduced, thereby improving the efficiency of the large model capability assessment method of this embodiment.

[0111] In a feasible embodiment, the capability dimension at least includes factual correctness and user needs, and step A30 includes steps A301 to A302:

[0112] Step A301, if the sub-score of factual correctness or user demand is lower than or equal to the preset minimum score, then based on the sub-scores of factual correctness and user demand, an evaluation score is obtained, and the evaluation score is lower than or equal to the preset minimum score;

[0113] It should be noted that the preset minimum score refers to the threshold for unsatisfactory answer quality. For example, on a 10-point scale, the preset minimum score can be set to 2 points. When it is lower than or equal to 2 points, it means that the answer quality is unsatisfactory.

[0114] It is understandable that factual correctness and user needs are the dimensions that have the greatest impact on the quality of model answer data among all dimensions. This is because when the model answer data does not meet factual correctness and user needs, it means that the model answer data does not provide any substantial help in answering the questions raised by the user. Therefore, when the sub-scores of factual correctness and user needs are both higher than the preset minimum scores, the model answer data has basic answer quality.

[0115] Step A302: If the sub-scores of factual correctness and user needs are higher than the preset minimum score, the sub-scores are averaged to obtain an evaluation score.

[0116] The large model capability assessment method of the present embodiment obtains sub-scores through sub-score assessments in multiple dimensions, and then averages the self-scores to obtain a comprehensive assessment score. The model answer data is evaluated from multiple dimensions, and the model answer data is a response to test data related to the field of government affairs circulation. Therefore, the large model capability assessment method of the present embodiment can improve the accuracy of the large model's capability assessment in the field of government affairs circulation.

[0117] In another feasible embodiment, the target scoring method includes a reference answer and a score calculation function. In step S20, the model answer data is scored according to the target scoring method, and the step of obtaining the evaluation score includes steps B10 to B20:

[0118] Step B10, inputting the reference answer and the model answer data into the score calculation function to calculate the sub-score corresponding to each ability dimension;

[0119] Step B20, calculating the average of the sub-scores to obtain an evaluation score.

[0120] In this embodiment, if the data volume of the model answer data is less than or equal to the preset data volume, the scoring method including the reference answer and the score calculation function is used as the target scoring method.

[0121] It should be noted that the score calculation function refers to a function for calculating sub-scores.

[0122] It can be understood that the score calculation function can be BLEU, ROUGE, etc. In this embodiment, rogue-L is used to calculate the score. First, the reference answer and model answer data are segmented; after segmentation, the LCS (Longest Common Subsequence) length between the reference answer and the model answer data is calculated; the recall rate recall = LCS length / length of the model answer data; the precision precision = LCS length / reference answer length; the f1 score = 2*(recall*precision) / (recall+precision) is calculated.

[0123] Furthermore, each dimension has a corresponding reference answer, so rogue-L can be used to calculate the sub-scores of each dimension, and then the sub-scores are averaged to obtain the evaluation score, thereby achieving a comprehensive evaluation of the capabilities of the preset reasoning model in the field of government affairs circulation.

[0124] The large model capability assessment method of the present embodiment can use rogue-L to implement a comprehensive assessment of the capability of the preset reasoning model in the field of government affairs circulation when the amount of model answer data is less than or equal to the preset data amount, intelligently identify the data scale of the model answer data, and implement rapid assessment of small-scale model answer data, thereby further improving the efficiency of the large model capability assessment method of the present embodiment.

[0125] Step S30: output the evaluation score and form score data together with the test data.

[0126] In a feasible embodiment, step S30 further includes step S301:

[0127] Step S301: store the test data in the score data in a preset corpus, and fine-tune the preset corpus based on the evaluation scores in the score data.

[0128] It should be noted that the preset corpus is a large collection of texts. The preset corpus is used as the basis for research and application in natural language processing (NLP), computational linguistics, machine learning and other fields. The preset corpus contains four fine-tuning models to provide selection for the base model. The base model refers to a model that is pre-trained based on the preset corpus, and the fine-tuning model is a model that is additionally trained on the basis of the base model for specific tasks or data sets.

[0129] It can be understood that, in this embodiment, the test data is placed into a preset corpus, and the evaluation score is input into the fine-tuning model for use as a basis for selecting the base model.

[0130] The large-model capability assessment method of this application assists in the construction of a preset corpus in the field of government affairs circulation by associating the test data and assessment scores of the capability assessment with a preset corpus, thereby improving the data quality in the preset corpus.

[0131] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 2 Before the step of performing reasoning operations on the test data by using a preset reasoning model to obtain model answer data, the large model capability evaluation method further includes steps S01 to S05:

[0132] Step S01, obtaining an original data set;

[0133] It should be noted that the original data set refers to a data set containing a large amount of sample data, which can come from various sources, such as Internet crawling, sensor collection, etc. Each sample contains multiple feature values ​​and corresponding label information.

[0134] It is understandable that the original data set contains data related to the field of government affairs circulation, such as the government affairs circulation system announced by the government and reports related to government affairs circulation.

[0135] Step S02, determining whether the original data set is an open source evaluation set, where the open source evaluation set is a data set that has been publicly released and is allowed to be freely downloaded and used;

[0136] It should be noted that open source evaluation sets refer to data sets that have been publicly released and allowed to be freely downloaded and used, that is, test data that has been publicly released and allowed to be freely downloaded and used.

[0137] Step S03: if the original data set is not an open source evaluation set, the original data set is cleaned to obtain cleaned data;

[0138] It should be noted that cleaned data refers to the original data set after cleaning.

[0139] It is understandable that for non-open source datasets, since they may have problems such as noise and missing values, they need to be cleaned first. The specific cleaning method can be flexibly selected according to the actual situation, such as deleting outliers, filling missing values, etc.

[0140] Step S04, inputting the cleaning data into a preset core module for processing to obtain test data;

[0141] It should be noted that the core module refers to the module used for generating test data in this embodiment.

[0142] It can be understood that the core module generates QA (Question Answering) for the cleaned data (books, papers, etc.), allowing the model to first generate summaries or keywords based on the cleaned data, then generate intermediate data through the cleaned data, and generate questions based on the intermediate data; then evaluate whether the questions are usable; generate answers to available questions by combining manual annotation with large models, and construct a QA question and answer evaluation data set (test data and reference answers).

[0143] Furthermore, by inputting the questions generated by QA into the preset reasoning model, model answer data can be obtained. By comparing with the reference answers, accurate evaluation of the capabilities of the large model can be achieved.

[0144] In a feasible embodiment, step S04 may include steps C10 to C40:

[0145] Step C10, generating key information data based on the cleansed data;

[0146] Step C20, generating question-answering questions based on key information data;

[0147] Step C30, determining whether the question-answering question is available, and classifying the question-answering question into available questions and unavailable questions;

[0148] Step C40, collect the available questions to form test data.

[0149] It should be noted that key information data refers to data extracted from cleaned data through annotation methods such as entity naming. Questions and answers are text content in the test data used to ask questions to the preset reasoning model. Available questions refer to questions related to the government affairs circulation field and logically correct, and unavailable questions refer to questions that are irrelevant to the government affairs circulation field or logically incorrect.

[0150] In a feasible embodiment, the open source model Llama-3-8B is used to automatically determine whether the question is available. The prompt instruction is as follows:

[0151] Prompt1=You are a question assessment expert who is proficient in Chinese. Your job is to determine whether the following questions are available. Please directly output "available" or "unavailable" without outputting any extra content.

[0152] The consistency rate is then used to determine whether the judgment of the Llama-3-8B model is aligned with that of humans. The calculation formula is:

[0153]

[0154] Among them, TP is a true positive example, that is, an available problem is predicted to be an available problem; TN is a true negative example, that is, an unavailable problem is predicted to be an unavailable problem; FP is a false positive example, that is, an unavailable problem is predicted to be an available problem; FN is a false negative example, that is, an available problem is predicted to be an unavailable problem.

[0155] After analysis, the judgment consistency rate of Llama-3-8B is as high as 99%, which can be considered to be able to replace manual work and complete automated judgment on whether the data is available.

[0156] Step S05: If the original data set is an open source evaluation set, the original data set is input into the core module for processing to obtain test data.

[0157] It is understandable that when the original data set is an open source evaluation set, as publicly used test data, there will basically be no problems such as noise and missing values. The data cleaning process can be skipped and directly input into the core module for processing to obtain test data.

[0158] The large model capability assessment method of this embodiment generates QA on data in the field of government affairs circulation, and then constructs test data. The large model capability assessment is performed through the above test data, which can comprehensively and deeply test the capabilities of the preset reasoning model in government affairs understanding, knowledge reasoning, etc.

[0159] For example, to help understand the technical concept or technical principle of the large model capability evaluation method after combining this embodiment with the above-mentioned embodiment 1 and embodiment 2, please refer to Figure 3 , Figure 3 A brief flow chart of a large model capability assessment method is provided, as follows:

[0160] The first step is to evaluate the data set: prepare an original data set containing a large amount of sample data. These samples can come from various sources, such as Internet crawling, sensor collection, etc. Each sample contains multiple feature values ​​and corresponding label information;

[0161] The second step is whether the evaluation set is open source: determine whether the original dataset is an open source evaluation set;

[0162] The third step is to directly normalize the original dataset if it is an open source evaluation set. If it is not an open source evaluation set, normalize it after data cleaning.

[0163] It can be understood that normalization refers to adjusting the data to a specific range or converting it into a standard form to facilitate comparison and analysis between different data. Normalization can facilitate comparison between the test data set and the reference answer when evaluating the capabilities of large models.

[0164] Step 4: Normalize the evaluation data set: Integrate the test data obtained by processing the open source evaluation set and the test data obtained by processing the original data set that is not an open source evaluation set into a normalized evaluation data set (test data);

[0165] Step 5: Input corpus: put the test data into the corpus;

[0166] Among them, the normalized evaluation data set is the test data, and the corpus is the preset corpus.

[0167] It should be noted that the above examples are only used to assist in understanding the present application and do not constitute a limitation on the large model capability assessment method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0168] The present application also provides a large model capability evaluation device, please refer to Figure 4 , the large model capability evaluation device comprises:

[0169] A model reasoning module 10 is used to perform reasoning operations on test data using a preset reasoning model to obtain model answer data, wherein the test data does not contain a data set with label information;

[0170] An evaluation module 20 is used to determine a target scoring method that matches the data volume of the model answer data, and score the model answer data according to the target scoring method to obtain an evaluation score, wherein the target scoring method scores the model answer data according to at least two capability dimensions;

[0171] The score output module 30 is used to output the evaluation score and form score data together with the test data.

[0172] In one embodiment, the target scoring method includes reference answers and scoring rules, and the evaluation module 20 is further used to:

[0173] Compare the model answer data with the reference answer, obtain the shortcomings of the model answer data and output an explanation of the shortcomings;

[0174] Based on the weakness explanation, scoring the model answer data by at least two capability dimensions to obtain a sub-score corresponding to each capability dimension;

[0175] According to the scoring rules, the sub-scores are comprehensively scored to obtain an evaluation score.

[0176] In one embodiment, the capability dimension at least includes factual correctness and user needs, and the evaluation module 20 is further used to:

[0177] If the sub-score of factual correctness or user needs is lower than or equal to the preset minimum score, an evaluation score is obtained based on the sub-scores of factual correctness and user needs, and the evaluation score is lower than or equal to the preset minimum score;

[0178] If the sub-scores of factual correctness and user needs are higher than the preset minimum score, the sub-scores are averaged to obtain an evaluation score.

[0179] In one embodiment, the target scoring method includes a reference answer and a score calculation function, and the evaluation module 20 is further used to:

[0180] Input the reference answer and the model answer data into the score calculation function to calculate the sub-score corresponding to each ability dimension;

[0181] The sub-scores are averaged to obtain an assessment score.

[0182] In one embodiment, the evaluation module 20 is further configured to:

[0183] Comparing the data volume of the model answer data with the preset data volume to obtain a comparison result;

[0184] A scoring method that matches the comparison result is determined from among a plurality of preset scoring methods as a target scoring method that matches the data volume of the model answer data.

[0185] In one embodiment, the large model capability evaluation device further includes:

[0186] Get the original data set;

[0187] Determine whether the original data set is an open source evaluation set, where the open source evaluation set is a data set that has been publicly released and allowed to be freely downloaded and used;

[0188] If the original data set is not an open source evaluation set, performing data cleaning on the original data set to obtain cleaned data;

[0189] Inputting the cleaning data into a preset core module for processing to obtain test data;

[0190] If the original data set is an open source evaluation set, the original data set is input into the core module for processing to obtain test data.

[0191] In one embodiment, the large model capability evaluation device further includes:

[0192] generating key information data based on the cleansed data;

[0193] Generate question-answering questions based on the key information data;

[0194] Determining whether the question-and-answer question is available, and dividing the question-and-answer question into available questions and unavailable questions;

[0195] The available questions are collected to form test data.

[0196] In one embodiment, the model reasoning module 10 is further used for:

[0197] Performing an inference operation on the logic capability data set through a preset inference model to obtain first model answer data, wherein the first model answer data is used to score the general capability;

[0198] The government affairs circulation data set is subjected to inference operation through a preset inference model to obtain second model answer data, and the second model answer data is used to score the government affairs circulation capability.

[0199] In one embodiment, the large model capability evaluation device further includes:

[0200] The test data in the score data are stored in a preset corpus, and the preset corpus is fine-tuned based on the evaluation scores in the score data.

[0201] The large model capability assessment device provided in the embodiment of the present application adopts the large model capability assessment method in the above embodiment, which can solve the technical problem of how to improve the accuracy of the large model capability assessment in the field of government affairs circulation. Compared with the prior art, the beneficial effects of the large model capability assessment device provided in the embodiment of the present application are the same as the beneficial effects of the large model capability assessment method provided in the above embodiment, and the other technical features in the large model capability assessment device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0202] The present application provides a large model capability assessment device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the large model capability assessment method in the above-mentioned embodiment one.

[0203] Reference below Figure 5 , which shows a schematic diagram of the structure of a large model capability evaluation device suitable for implementing the embodiment of the present application. The large model capability evaluation device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The large model capability assessment device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0204] like Figure 5 As shown, the large model capability evaluation device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the large model capability evaluation device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the large model capability assessment device to communicate with other devices wirelessly or by wire to exchange data. Although the diagram shows a large model capability assessment device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0205] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0206] The large model capability assessment device provided by this application adopts the large model capability assessment method in the above embodiment, which can solve the technical problem of how to improve the accuracy of large model capability assessment in the field of government affairs circulation. Compared with the prior art, the beneficial effects of the large model capability assessment device provided by this application are the same as the beneficial effects of the large model capability assessment method provided by the above embodiment, and the other technical features in the large model capability assessment device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0207] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0208] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0209] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the large model capability assessment method in the above-mentioned embodiment.

[0210] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0211] The computer-readable storage medium may be included in the large model capability evaluation device; or may exist independently without being assembled into the large model capability evaluation device.

[0212] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the large model capability evaluation device, the large model capability evaluation device: performs inference operations on the test data through a preset inference model to obtain model answer data, and the test data does not contain a data set with label information; determines a target scoring method that matches the data volume of the model answer data, and scores the model answer data according to the target scoring method to obtain an evaluation score, wherein the target scoring method scores the model answer data through at least two capability dimensions; outputs the evaluation score and forms score data together with the test data.

[0213] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0214] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0215] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0216] The storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned large model capability evaluation method, and can solve the technical problem of how to improve the accuracy of large model capability evaluation in the field of government affairs circulation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the large model capability evaluation method provided in the above-mentioned embodiment, and will not be repeated here.

[0217] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. A large model capability assessment method, characterized in that: The large model capability evaluation method comprises: Performing inference operations on the test data using a preset inference model to obtain model answer data, wherein the test data does not include a data set with label information; Determining a target scoring method that matches the data volume of the model answer data, and scoring the model answer data according to the target scoring method to obtain an evaluation score, wherein the target scoring method scores the model answer data according to at least two capability dimensions; The evaluation score is output and forms score data together with the test data.

2. The method according to claim 1, characterized in that The target scoring method includes reference answers and scoring rules, and the step of scoring the model answer data according to the target scoring method to obtain an evaluation score includes: Compare the model answer data with the reference answer, obtain the shortcomings of the model answer data and output an explanation of the shortcomings; Based on the weakness explanation, scoring the model answer data by at least two capability dimensions to obtain a sub-score corresponding to each capability dimension; According to the scoring rules, the sub-scores are comprehensively scored to obtain an evaluation score.

3. The method according to claim 2, characterized in that The capability dimension at least includes factual correctness and user needs, and the step of comprehensively scoring the sub-scores according to the scoring rules to obtain an evaluation score further includes: If the sub-score of factual correctness or user needs is lower than or equal to the preset minimum score, an evaluation score is obtained based on the sub-scores of factual correctness and user needs, and the evaluation score is lower than or equal to the preset minimum score; If the sub-scores of factual correctness and user needs are higher than the preset minimum score, the sub-scores are averaged to obtain an evaluation score.

4. The method according to claim 1, characterized in that The target scoring method includes a reference answer and a score calculation function, and the step of scoring the model answer data according to the target scoring method to obtain an evaluation score includes: Input the reference answer and the model answer data into the score calculation function to calculate the sub-score corresponding to each ability dimension; The sub-scores are averaged to obtain an assessment score.

5. The method according to claim 1, characterized in that The step of determining a target scoring method that matches the data volume of the model answer data includes: Comparing the data volume of the model answer data with the preset data volume to obtain a comparison result; A scoring method that matches the comparison result is determined from among a plurality of preset scoring methods as a target scoring method that matches the data volume of the model answer data.

6. The method according to claim 1, characterized in that Before the step of performing reasoning operation on the test data by using the preset reasoning model to obtain model answer data, the method includes: Get the original data set; Determine whether the original data set is an open source evaluation set, where the open source evaluation set is a data set that has been publicly released and allowed to be freely downloaded and used; If the original data set is not an open source evaluation set, performing data cleaning on the original data set to obtain cleaned data; Inputting the cleaning data into a preset core module for processing to obtain test data; If the original data set is an open source evaluation set, the original data set is input into the core module for processing to obtain test data.

7. The method according to claim 6, characterized in that The step of inputting the cleaning data into a preset core module for processing to obtain test data comprises: generating key information data based on the cleansed data; Generate question-answering questions based on the key information data; Determining whether the question-and-answer question is available, and dividing the question-and-answer question into available questions and unavailable questions; The available questions are collected to form test data.

8. The method according to claim 1, characterized in that The test data includes a logic capability data set and a government affairs flow data set, and the model answer data includes first model answer data and second model answer data; The step of performing an inference operation on the test data by using a preset inference model to obtain model answer data includes: Performing an inference operation on the logic capability data set through a preset inference model to obtain first model answer data, wherein the first model answer data is used to score the general capability; The government affairs circulation data set is subjected to inference operation through a preset inference model to obtain second model answer data, and the second model answer data is used to score the government affairs circulation capability.

9. The method according to claim 1, characterized in that After the step of outputting the evaluation score and forming the score data together with the test data, the method further comprises: The test data in the score data are stored in a preset corpus, and the preset corpus is fine-tuned based on the evaluation scores in the score data.

10. A large model capability assessment device, characterized in that: The large model capability assessment device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the large model capability assessment method according to any one of claims 1 to 9.

11. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the large model capability assessment method as described in any one of claims 1 to 9 are implemented.

Images