Finance and tax field data set evaluation method based on OpenCompass

Abstract

The invention discloses a finance and taxation field data set evaluation method based on OpenCompass. The method comprises the steps that S1, finance and taxation model interface parameters are configured for an OpenCompass evaluation tool; s2, setting a finance and tax data set and an automatic evaluation overall process thereof; s3: according to the type of the question in the data set result, establishing a cue word of the finance and taxation model; s4, performing formatting processing on answer results in the finance and tax model data set; and S5, based on an OpenCompass evaluation tool, evaluating finance and tax model data set results one by one according to the content of the model cue word. Therefore, the effect of automatically evaluating the training model is improved, and the manual auditing cost after a large number of data set models are answered is reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for evaluating a data set in the finance and taxation field based on OpenCompass. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) are increasingly used in various fields. OpenCompass is a one-stop large model evaluation platform that supports the evaluation of multiple models and data sets. Its features include support for distributed and efficient evaluation, diversified evaluation paradigms, modular design and strong scalability, as well as a complete experimental management and reporting mechanism. These features enable OpenCompass to efficiently and comprehensively evaluate the capabilities of large models in various fields. However, the main data sets of the OpenCompass evaluation system are the Huggingface data set and third-party data sets. However, how to accurately and efficiently evaluate the performance of models in the field of finance and taxation has become an urgent problem to be solved.

[0003] In the prior art, for example, CN117785664A provides an automated large model testing system and method of different dimensions and directions. This method mainly classifies and analyzes the data input by the user in the step of the data acquisition unit and converts it according to the data requirement format to obtain formatted data. Finally, the formatted data is evaluated absolutely and relatively to obtain the evaluation results and output the data. Although this evaluation method has multiple dimensions and angles, most dimensions are highly subjective. In the results of the evaluation of the sorted data set, it may be highly disruptive due to the splitting of different dimensions. Another example is CN117911179A, which provides a method for calculating the attribute characteristic value of fiscal and tax data and at least one verification data corresponding to the audited fiscal and tax data, thereby obtaining the audit results of the fiscal and tax data to be audited, so as to facilitate enterprises and fiscal and tax users to quickly discover and correct problems with fiscal and tax data and shorten the time cost of fiscal and tax calculations. Although this method can effectively identify and correct problems in financial and tax calculations, due to the diversity of data, the constraints of at least one acceptance data corresponding to the data require higher requirements, and the scalability is relatively low. Summary of the invention

[0004] To solve the above problems, the present invention provides a method for evaluating a data set in the field of finance and taxation based on OpenCompass, which uses a custom data set for automatic evaluation. The financial and taxation data set is evaluated and scored from the aspects of model parameter configuration, financial and taxation data set matching, model prompt words, decision rules, log analysis, support for subjectivity and objectivity, etc., which can meet the needs of different data in the field of finance and taxation and improve the diversity of data sets. Due to the specificity of the matching rules, different rules can be used to verify subjective and objective questions by category, which further improves the accuracy of the model results. Through the constraints of the model prompt words, the answer results of the model are output according to the requirements of the prompt words, and the consistency and accuracy of the model results are enhanced. Finally, by setting the scoring rules of different dimensions, the model answer results are scored by category, and finally the final score of the data is judged according to different weights. The final model outputs key information such as model results and scoring data.

[0005] To achieve the above objectives, the present invention provides a method for evaluating a data set in the field of finance and taxation based on OpenCompass, which comprises:

[0006] Step S1: Configure the financial and tax model interface parameters for the OpenCompass evaluation tool;

[0007] Step S2: Set up the financial and taxation data set and its overall automated evaluation process, including:

[0008] Step S201: According to the finance and taxation field, finance and taxation related questions that have been manually classified according to rules in advance and their reference answers are used to form a finance and taxation data set;

[0009] Step S202: inputting the fiscal and taxation data set into the fiscal and taxation model to obtain the answer result of the fiscal and taxation model;

[0010] Step S203: format the answer results of the fiscal and taxation model according to the classification requirements and output the final data set results;

[0011] Step S204: scoring the data set results outputted in step S203 according to preset evaluation rules;

[0012] Step S205: Output the data set results and their corresponding analysis.

[0013] Step S3: Establish prompt words for the financial and taxation model according to the type of questions in the data set results;

[0014] Step S4: formatting the answer results in the tax model data set to obtain answer results of different data categories;

[0015] Step S5: Based on the OpenCompass evaluation tool and according to the content of the model prompt words, the results of the financial and tax model data set are evaluated one by one.

[0016] In one embodiment of the present invention, step S1 specifically includes:

[0017] Step S101: Create a fiscal and taxation model file in the models folder, configure the method for retrieving fiscal and taxation model interface parameters in the fiscal and taxation model file, and then configure the model request parameters; wherein the model request parameters include the fiscal and taxation model custom class name, model name, access permission token, and model address;

[0018] Step S102: adjusting the temperature coefficient required by the tax model data set, generating the highest cumulative probability parameter of text quality and the model constraint parameter;

[0019] Step S103: Set the maximum number of generated tokens and the maximum input length.

[0020] In one embodiment of the present invention, during the transmission of the fiscal and taxation data set in step S202, corresponding fiscal and taxation classification fields are set for different data classifications, and the value of each fiscal and taxation classification field is matched with the model prompt word through a dictionary to find the model prompt word of the corresponding classification question; then the fiscal and taxation model finds the corresponding model prompt word according to the classification of the data to carry out problem thinking and generate answers to normative questions; finally, the answer result of the fiscal and taxation model is provided for the input fiscal and taxation data set.

[0021] In one embodiment of the present invention, step S203 specifically includes: firstly screening the answer results of the financial and taxation model for data validity according to the classification requirements, then inserting segment breaks according to the characteristics of the results of the financial and taxation calculations and accounting entry questions, forming formatted JSON data, and then outputting the final data set results, wherein the data set results include: classified questions, reference answers to the questions, and the answers to the questions by the financial and taxation model.

[0022] In one embodiment of the present invention, the formatting process of the accounting entry field in step S4 is specifically as follows:

[0023] Step S401: Screen the answer results in the accounting entry field in the tax model data set and divide them into concept questions, accounting entry questions without calculations, and accounting entry questions with calculations;

[0024] Step S402: For concept questions, obtain valid content through preset identifiers, and then format non-text characters and line breaks through the invalidFilter function to output the content that needs to be retained;

[0025] For accounting entry questions without calculations, first execute the processing of concept questions, then use the pattern_accounting function to format the debit and credit relationships in the answer and output the processing results;

[0026] For accounting entry questions that require calculations, first map and match the implicit tax rate keywords with the tax rate taxMap object, then format the calculation process into separate calculation steps, and then execute the processing of accounting entry questions that do not require calculations.

[0027] In one embodiment of the present invention, the formatting process of step S4 for the tax field is specifically as follows:

[0028] Step S411: Screen the answer results in the taxation field in the fiscal and taxation model data set and divide them into basic policy type questions and calculation type questions;

[0029] Step S412: For basic policy type questions, obtain valid content through preset identifiers, and then format non-text characters and line breaks through the invalidFilter function to output the content that needs to be retained;

[0030] For calculation type questions, first map and match the implicit tax rate keywords with the tax rate taxMap object, then format the calculation process into calculation steps, and then execute the processing of basic policy type questions. Among them, when the data set contains Chinese numerals, it is also necessary to convert the text and numbers.

[0031] In one embodiment of the present invention, step S5 specifically includes:

[0032] Step S501: Compare and judge the reference answer to the question and the answer of the fiscal and taxation model from different dimensions. If the consistency between the two reaches the preset threshold standard, it is determined to be a correct answer, and the corresponding question is rated with a star rating, and the reason for the rating is given; for the answers that are determined to be inconsistent, the inconsistent content parts are marked, and the error cause analysis and star rating are provided;

[0033] Step S502: Output the referee result according to a preset format.

[0034] In one embodiment of the present invention, the part of the content identified as inconsistent in step S501 and providing error cause analysis is specifically: the inconsistent content is identified and extracted from the result of the financial and tax model answer, and the explanation of the corresponding part of the reference answer is used as the reason for the wrong model answer.

[0035] In one embodiment of the present invention, the process of outputting the judgment result in step S502 is specifically as follows:

[0036] Step S5021: traverse all json data files in the input directory;

[0037] Step S5022: formatting the answers of the fiscal and taxation model to different categories of data in the data set according to the requirements of different categories of answers;

[0038] Step S5023: Judge the results of the tax model and the reference answers, and judge them one by one in different dimensions and steps according to the prompt word rules;

[0039] Step S5024: normalize the number of stars in different dimensions in the results after judging according to the weights of the different dimensions;

[0040] Step S5025: Map and output the normalized results in the form of "partially correct, correct, wrong", and perform statistics on the data set results;

[0041] Step S5026: Convert the data processed by the above steps and the detailed data field results into binary format and output them in xlsx format.

[0042] Compared with the prior art, the method for evaluating a data set in the finance and taxation field based on OpenCompass provided by the present invention improves the effect of automatic evaluation of training models and reduces the cost of manual review after answering a large number of data set models. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A schematic diagram of a flow chart of an embodiment of the present invention;

[0045] Figure 2 Schematic diagram of a data set result evaluation rule table according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Figure 1 FIG. 1 is a flow chart of an embodiment of the present invention. Figure 1 As shown, this embodiment provides a method for evaluating a data set in the financial and taxation field based on OpenCompass, which includes:

[0048] Step S1: Configure the financial and tax model interface parameters for the OpenCompass evaluation tool;

[0049] In this embodiment, step S1 specifically includes:

[0050] Step S101: Create a fiscal and taxation model file in the models folder. This embodiment takes the fiscal and taxation model file as FisTaxModel as an example. Configure the method for calling the fiscal and taxation model interface parameters in the fiscal and taxation model file. This embodiment takes the fiscal and taxation model interface parameter method as the FinancialTax method as an example. Then configure the model request parameters to call the corresponding fiscal and taxation model interface; wherein the model request parameters include the fiscal and taxation model custom class name (such as type:FisTax), the model name (such as path:str), the access permission token (such as api_key:str) and the model address (url:str). The fiscal and taxation model file of this embodiment reads the model parameter name corresponding to the type in the fiscal and taxation model through the FinancialTax method, thereby maintaining the synchronization of the model interface parameters;

[0051] Step S102: In addition to the above-mentioned model description parameters and API path parameters and other request information, it is also necessary to adjust the temperature coefficient (temperature) required by the tax model data set, the highest cumulative probability parameter (top_k) for generating text quality, and the model constraint parameter (top_p, based on the candidate word quantity constraint model in the tax knowledge base) to meet the data results; in this embodiment, since the result of each answer of the tax model needs to be consistent with human thinking, the threshold of the temperature coefficient temperature belonging to the tax field is configured to be 0.0, so that each answer of the model is within the scope of the knowledge point and has a variety of answers;

[0052] Step S103: Set the maximum number of generated tokens (max_out_len) and the maximum input length (max_seq_len); This embodiment needs to take into account the performance issues when the model dataset is running, and provide a numerical range that can correctly answer and explain the question while ensuring that the user accepts the time to wait for the answer to the question. Therefore, the maximum number of generated tokens max_out_len and the maximum input length max_seq_len are both set to 1024 to optimize the final answer result of the model.

[0053] For the sake of clarity, this embodiment uses the following parameter configuration code as an example to illustrate the above configuration process, but is not limited thereto:

[0054]

[0055] This embodiment uses the FinancialTax method to configure the optional parameters required by the financial and tax model in the OpenCompass evaluation tool, and makes a request through the type attribute value. According to the diversity of answers to financial and tax model questions and the performance required by the model data set answer results, the temperature coefficient and maximum input and output length that conform to the financial and tax data set are set. In this data set in a specific professional field, the temperature coefficient parameters unique to the model are formulated through the professional data set results, and the maximum input length and model output length are controlled, which improves the accuracy of the data set results and the performance of the model, thereby achieving an effective model answer result.

[0056] Step S2: Set up the financial and taxation data set and its overall automated evaluation process, including:

[0057] Step S201: According to the finance and taxation field, finance and taxation related questions that have been manually classified according to the rules and their reference answers are used to form a finance and taxation data set; thus, this embodiment has finance and taxation related business questions and the standard answers corresponding to the questions, which is convenient for benchmarking in subsequent evaluations;

[0058] Step S202: inputting the fiscal and taxation data set into the fiscal and taxation model to obtain the answer result of the fiscal and taxation model;

[0059] In this embodiment, during the transmission of the financial and taxation data set in step S202, corresponding financial and taxation classification fields (financialCategory) are set for different data classifications, and the value of each financial and taxation classification field is matched with the model prompt word through a dictionary to find the model prompt word of the corresponding classification question; then the financial and taxation model finds the corresponding model prompt word according to the classification of the data, so as to think about the problem and generate answers to normative questions, and finally provides the answer results of the financial and taxation model for the input financial and taxation data set.

[0060] Step S203: format the answer results of the fiscal and taxation model according to the classification requirements and output the final data set results;

[0061] In this embodiment, step S203 specifically involves first screening the answer results of the financial and taxation model for data validity according to the classification requirements, such as excluding answers with infinite loops of certain words, and then inserting paragraph breaks according to the characteristics of the results of the financial and taxation calculations and accounting entry questions, forming formatted JSON data and then outputting the final data set results, wherein the data set results include: classified questions, reference answers to the questions, and the answers to the questions by the financial and taxation model.

[0062] Step S204: scoring the data set results outputted in step S203 according to preset evaluation rules;

[0063] Figure 2 FIG. 1 is a schematic diagram of a data set result evaluation rule table according to an embodiment of the present invention. Figure 2 As shown, in this embodiment, the scoring rules may include first scoring the accuracy, completeness and coherence of the results separately, and then obtaining a comprehensive score by combining the scores of the three aspects to obtain a scoring result. The specific scoring rules can be pre-set according to specific needs, and the present invention does not limit them.

[0064] Step S205: Output the data set results and their corresponding analysis. The output data set results of this embodiment include the questions in the data set, the corresponding reference answers, the corresponding model answer results, and the scoring instructions and conclusions of the questions in terms of accuracy, completeness, and coherence; the corresponding analysis includes the scoring results of the fiscal and taxation model on different dimensions of different categories of data sets; the ability to score in different dimensions; and the scoring results of different dimensions are multiplied by coefficients 0.4, 0.4, and 0.2 respectively, and then converted into the final accuracy rate under the question, where the accuracy rate is not less than 0.5 is correct, and the rest is wrong. The above analysis is only exemplary and not restrictive.

[0065] Step S3: Establish prompt words for the financial and taxation model according to the type of questions in the data set results;

[0066] For different types of data, the prompt words answered by the finance and taxation model will inevitably be different. Since the present invention is used for data sets in the finance and taxation field, the answers to the questions need to be limited to the finance and taxation field. In addition, due to different questions, the results given by the finance and taxation model are also required to be specific, so different prompt words need to be given for different types of questions. In this embodiment, step S3 specifically classifies the questions in the data set results by calling the classification model, and assembles corresponding prompt words (prompt) according to the type of results after classification; or when the user enters a question, selects which classification category the question belongs to, and then directly assembles the prompt words according to the category selected by the user.

[0067] In one embodiment of the present invention, taking the accounting entry field as an example, the corresponding prompt words are displayed as follows:

[0068] Prompt="'You are a business expert in the field of finance and taxation. Your job is to evaluate the answers of the AI ​​assistant and the reference answers. Your evaluation needs to consider four aspects: answer accuracy, answer relevance, answer completeness, and sentence coherence. When judging, you need to follow the following requirements:

[0069] First, get answers to fiscal and tax model questions

[0070] Secondly, compare the answers to the fiscal and taxation model with the reference answers. Please use objective standards to judge the accuracy and defects of the answers to the fiscal and taxation model.

[0071] Next, we will rate the answers of the tax model from four aspects of the tax field, namely, answer accuracy, answer relevance, answer completeness, and sentence coherence. We will also explain the reasons for the star rating results in each aspect and give a specific number of stars. The number of stars in each dimension is an integer. One star represents complete dissatisfaction and five stars represent very satisfaction.

[0072] Finally, based on the scoring rules of the above dimensions, the final total score is given according to the coefficient of each dimension. The total score has only three gradients [0, 0.5, 1]. The output format of the referee result is json format as follows: {'Answer accuracy': 4 stars, 'Answer relevance': 4 stars, 'Answer completeness': 4 stars, 'Sentence coherence': 5 stars, 'Comprehensive score': 1}.

[0073] In another embodiment of the present invention, taking the finance and taxation field as an example, the corresponding prompt words are displayed as follows:

[0074] [Financial and tax issues]

[0075] A high-tech enterprise (not a small and micro-profit enterprise) that is supported by the state has sales of 80,000, 100,000, one hundred and sixty thousand yuan in the first quarter of 2024. Assuming that factors such as cost and expenses are not considered, and corporate income tax is paid in full on sales, please calculate the corporate income tax payable by the company and prepare the corresponding accounting entries.

[0076] [Financial and tax issues]

[0077] [Standard answer]

[0078] Steps to solve the problem:

[0079] Calculate taxes: First, calculate the taxes to be paid. Tax = sales × tax rate = 340,000 × 15% = 51,000 yuan.

[0080] Prepare accounting entries:

[0081] Debit: Income tax expense 51,000 yuan

[0082] Credit: Taxes payable - Corporate income tax payable RMB 51,000

[0083] Debit: Taxes payable - corporate income tax payable 51,000 yuan

[0084] Credit: Bank deposit / cash on hand RMB 51,000

[0085] The enterprise income tax rate applicable to high-tech enterprises that are key supported by the state is 15%. This policy is mainly based on Article 28 of the Enterprise Income Tax Law of the People's Republic of China, which clearly stipulates that high-tech enterprises that the state needs to focus on supporting shall be subject to a reduced enterprise income tax rate of 15%.

[0086] [Standard answer]

[0087] [Financial and tax model answer]

[0088] First, we need to calculate the taxable income of the high-tech enterprise in the first quarter. Since the title mentions that the sales amount is fully subject to corporate income tax, and does not consider factors such as cost and expenses, the taxable income is the sales amount.

[0089] Sales in the first quarter were RMB 80,000, RMB 100,000 and RMB 160,000 respectively, totaling RMB 340,000.

[0090] Next, according to the second paragraph of Article 28 of the Enterprise Income Tax Law of the People's Republic of China, high-tech enterprises that are key supported by the state shall be subject to a reduced enterprise income tax rate of 15%. Therefore, the enterprise income tax payable is:

[0091] Corporate income tax payable = taxable income × tax rate

[0092] =340,000 yuan×15%

[0093] =51,000 yuan

[0094] Finally, we can prepare the corresponding accounting entries:

[0095] Debit: Income tax expense RMB 51,000

[0096] Loan: Taxes payable - Corporate income tax payable: RMB 51,000

[0097] In this way, we have completed the calculation of the corporate income tax payable by the high-tech enterprise in the first quarter and prepared the corresponding accounting entries.

[0098] [Financial and tax model answer]

[0099] First, we compare the answer of the finance and taxation model with the reference answer. In describing how to calculate the corporate income tax payable by the company and the corresponding accounting entries, first of all, the answer of the finance and taxation model provides detailed steps, including the total sales amount in the first quarter, the corporate income tax rate, and notes the conversion of the English numerals of the sales amount into Arabic numerals. At the same time, the difference between the corporate income tax rate corresponding to high-tech enterprises and the corporate income tax rate of non-high-tech enterprises is also translated. That is: a corporate income tax rate of 15%, which is correct. Secondly, in accounting entries, the debit and credit order and accounts can be used correctly, and the numbers are correct.

[0100] Now, we evaluate the answers of the fiscal and taxation model according to different dimensions:

[0101] 1. Answer accuracy: The answer of the fiscal and taxation model is correct in the translation of the sales figures in the first quarter. Secondly, the total sales in the first quarter and the use of the corporate income tax rate unique to high-tech enterprises are also correct. However, when making the accounting entries at the end, it is not clear whether it is the accounting entry at the time of provision or the accounting entry for the actual payment of corporate income tax. Therefore, there are some flaws in accuracy. Therefore, in this dimension, the answer of the fiscal and taxation model is flawed, and the number of stars given is 4.

[0102] 2. Answer relevance: The user's question is how to calculate the corporate income tax payable by the company and the corresponding accounting entries. The answer of the fiscal and tax model provides the calculation steps and results of corporate income tax, and also gives the preparation of accounting entries. Therefore, the answer of the fiscal and tax model is completely relevant and is given 5 stars.

[0103] 3. Sentence coherence: The answer of the fiscal and taxation model is coherent as a whole. From calculating the total sales in the first quarter to the implicit corporate income tax rate of 15% for high-tech enterprises, and then to deriving the tax payable in the first quarter, the steps are logically coherent. The order of debit and credit entries and the use of accounts are also coherent. Therefore, the sentence coherence of the fiscal and taxation model is rated 5 stars.

[0104] 4. Completeness of answer: The answer of the fiscal and taxation model provides detailed steps for calculating corporate income tax, but in the final step of making accounting entries, it does not list all the accounting entries for accruing corporate income tax and actually paying corporate income tax, resulting in an incomplete answer. Therefore, the fiscal and taxation model is given 4 stars for completeness of answer.

[0105] Based on the above analysis, the answer of the fiscal and taxation model is correct in terms of the steps of corporate income tax calculation, but the corporate income tax accounting entries given at the end are not comprehensive. Therefore, I will give the answer of the fiscal and taxation model a higher score. The specific stars are as follows:

[0106] *Answer accuracy: 4 (Since the accounting entry given at the end did not specify which situation it was, one star was deducted in this dimension, giving a total of 4 stars)

[0107] *Answer relevance: 5 (the answer of the fiscal and tax model meets the user's needs, so it received 5 stars)

[0108] * Sentence coherence: 5 (The answer to the fiscal and tax model is coherent overall, with correct format and orderly paragraphs, so it received 5 stars)

[0109] *Answer completeness: 4 (Since the final corporate income tax accounting entry did not clearly indicate whether it was an accounting entry at the time of provision or at the time of payment, the answer was not complete, so one star was deducted, and the total score was 4 stars)

[0110] Combining the number of stars in the above dimensions and according to the weight coefficient of each dimension, we can get (score1*w1+score1*w1+score1*w1+score1*w1) / 5=0.88. From the three gradients of the total score [0,0.5,1], the comprehensive score I gave to the answer of the fiscal and tax model is 1. Although the accounting entries for different situations are not given in detail in the accounting entries, on the whole, its answer is correct, meeting the requirements of answer relevance, sentence coherence, answer completeness and providing complete information.

[0111] Finally, the scoring result is returned in the required dictionary format:

[0112] ```json

[0113] {'Answer accuracy':4,'Answer relevance':5,'Sentence coherence':5,'Answer completeness':4,'Overall score':1}.

[0114] The above embodiment is only used to illustrate a certain prompt word, and is not used to limit all prompt words of the present invention. The setting of specific prompt words can be set according to the problem classification and its evaluation requirements.

[0115] The automated evaluation of the present invention is mainly to reduce labor costs. Its key issue is to achieve automation, so it is necessary to constrain the refereeing rules of the OpenCompass evaluation tool. The key point of this embodiment is to let the OpenCompass evaluation tool answer according to the preset constraint rules, analyze the knowledge points in the professional questions in the data set through the OpenCompass evaluation tool, and then verify the results of the fiscal and taxation model and the reference answers of the data set according to the rules of the prompt words, and comprehensively examine the accuracy of the fiscal and taxation model to answer the questions, the professionalism of the knowledge, the completeness of the answer results and the coherence of the answer process through the four aspects of accuracy, relevance, coherence and integrity. Through the matching of these four dimensions, find out the difference between the results of the fiscal and taxation model and the standard answer. In order to perform objective scoring, give the correct content and error reasons of the model answer, and give the objective number of stars in different dimensions. This rule of constraining the answer and the dimension of the answer can be used to more comprehensively rate the professional model answer results. At the same time, in terms of evaluation integrity, the details of the fiscal and taxation model answer questions can be accurately captured, and the reasons for the incorrect identification of the fiscal and taxation model can be found, which is convenient for subsequent analysis and self-optimization.

[0116] Step S4: formatting the answer results in the tax model data set to obtain answer results of different data categories;

[0117] In this embodiment, the formatting process of step S4 for the accounting entry field is specifically as follows:

[0118] Step S401: Screen the answer results in the accounting entry field in the tax model data set, and divide them into concept questions, accounting entry questions without calculations, and accounting entry questions with calculations. In accounting entries, due to the specificity of answers to professional questions, the results need to be screened;

[0119] Step S402: For concept questions, obtain valid content through a preset identifier (e.g., answer is), and then format non-text characters and line breaks through the invalidFilter function to output the content that needs to be retained;

[0120] For accounting entry questions without calculations, first execute the processing of concept questions, then format the debit and credit relationship in the answer through the pattern_accounting function and output the processing result; the pattern_accounting function is a custom function that can filter out the debit and credit part of the answer and format it into the accounting entry writing format, which can include the entry form, the format indentation and line break of the entry directory and the sub-directories at different levels, as well as the entry content, so as to meet the writing requirements of accountants;

[0121] For accounting entry questions that require calculations, first map and match the implicit tax rate keywords with the tax rate taxMap object, then format the calculation process into separate calculation steps, and then execute the processing of accounting entry questions that do not require calculations.

[0122] Among the results of the three categories of questions, the results of the concept questions do not involve the production of entries, but are generally just the explanation and application of basic concepts. Therefore, for the answers to the questions in the data set of the basic concept questions, first use the answer is identifier to obtain the valid content, then use the invalidFilter function to format non-text characters and line breaks, and then output the content that needs to be retained after processing. For accounting entries without calculations, the answers to general questions involve explanations of relevant conceptual content and the debit / credit format content unique to the corresponding accounting entries. Therefore, in this type of question, in addition to the same processing steps as the conceptual question, it is also necessary to format the debit and credit entries in the accounting entries. Here, a pattern_accounting function is needed to format the debit and credit relationships in the answer to the question. On the one hand, it is convenient to give results in the dimension of automated assessment accuracy, and on the other hand, it is clearer to evaluate the results of the dimension of consistency. For accounting entries with calculations, the processing process is more complicated because it involves accounting-related professional knowledge and calculation rules, as well as the production of accounting entries. In order to obtain a more detailed analysis of the problem, the pattern_accountingentry function can be used to define rules for stripping and extraction. First, it is necessary to map and match the implicit tax rate keyword part, such as "high-tech enterprises / small and micro enterprises / small-scale taxpayers / environmental protection / agriculture, finance", etc., to match the corresponding implicit tax rate keyword, and format the calculation process in calculation steps. The purpose is to make the model results more accurate and consistent, and then proceed with the processing of accounting entries without calculations. The pattern_accountingentry function is a custom function, which aims to distribute and execute the accounting entry data set with calculation steps according to the calculation priority, so as to format the answer into a clear decomposition and step-by-step scoring effect.

[0123] In this embodiment, the formatting process for the tax field in step S4 is specifically as follows:

[0124] Step S411: Screen the answer results in the tax field in the fiscal and tax model dataset, which are divided into basic policy type questions and calculation type questions;

[0125] Step S412: For basic policy type questions, obtain the valid content through a preset identifier (such as "answer is"), and then perform formatting processing on non-alphanumeric characters and line breaks through the invalidFilter function to output the content to be retained;

[0126] For calculation type questions, first perform mapping and matching of the taxMap object for the part containing implicit tax rate keywords, then format the calculation process by calculation steps, and then execute the processing process for basic policy type questions. Among them, when the dataset contains Chinese numeral words, text and number conversion is also required.

[0127] Since the dataset in the tax field also involves unit conversion and proper nouns containing hidden tax rates, the model results for the tax rate dataset also need to be subjected to the same operations as those with calculation part data in accounting entries. That is, for words containing tax rates in the dataset, such as "high-tech enterprises / small and micro enterprises / small-scale taxpayers / environmental protection / agriculture, finance", the same mapping is first performed to convert them into numbers that can be recognized by the computer; secondly, for the text containing Chinese numerals in the dataset, such as "one, two, three, four, five, six, seven, eight, nine, ten, hundred, thousand, ten thousand, hundred million", text and number conversion is performed to convert them into the same order of magnitude of the smallest unit, which is convenient for use in subsequent calculations and steps, and the model answer is more accurate. Therefore, in terms of the question types in the tax field, they are mainly divided into two categories: basic policy type and calculation type. In this embodiment, it is mainly necessary to format the data in the answers to calculation type questions in the tax field to finally obtain accurate model answer results.

[0128] The answer of the fiscal and taxation model to the data set question is in the format of a long string. Since the fiscal and taxation model of this embodiment will perform different matching rule checks for the two major categories of accounting entries and fiscal and taxation calculations. For accounting entries, the key is that the accounting entries will involve basic concept explanations, accounting entries without calculations, and accounting entries with implicit tax rate calculations. Since the automated assessment has a completeness dimension, it is necessary to match the rules of the debit and credit relationships and entry subjects in the accounting entry data set and display them separately. This is also the key content of the model answer in the accounting entry data set. For data sets of this type of implicit tax rates and Chinese numbers in capital letters, since tax rate calculations are involved, the steps of the fiscal and taxation model answer need to be formatted. The focus is on whether the model has correctly converted the implicit tax rate and the minimum unit conversion of the data unit magnitude during the data set answer process. By formatting the answer result of the fiscal and taxation model, it is formatted content with a complete set of answers, such as professional knowledge analysis, conversion of Chinese digital descriptions, correct writing of entry subjects and debit and credit relationships, and multi-step calculation processes in related tax rates. During the final judging process, the OpenCompass evaluation tool can give a more detailed and objective number of stars, and the results are close to those of manual scoring, ensuring the usability of automated evaluation.

[0129] Step S5: Based on the OpenCompass evaluation tool and according to the content of the model prompt words, the results of the financial and tax model data set are evaluated one by one.

[0130] In this embodiment, step S5 specifically includes:

[0131] Step S501: Compare and judge the reference answer to the question and the answer of the fiscal and taxation model from different dimensions. If the consistency between the two reaches the preset threshold standard, it is determined to be a correct answer, and the corresponding question is rated with a star rating, and the reason for the rating is given; for the answers that are determined to be inconsistent, the inconsistent content parts are marked, and the error cause analysis and star rating are provided;

[0132] In one embodiment of the present invention, the accounting entry field has only the dimensions of accuracy, completeness and consistency, and the tax field has the dimensions of accuracy, relevance, completeness and consistency; and the preset threshold on each dimension can be set to 3 points, for example, when the score is greater than 3 points, it is a correct answer, and other assessments can be, for example, equal to 3 points for partially correct, and less than 3 points for incorrect. This embodiment is only used for illustration and is not limited to this.

[0133] In this embodiment, the part of the content identified as inconsistent in step S501 and providing error cause analysis specifically includes: identifying and extracting inconsistent content in the result of the financial and tax model answer, and using the explanation of the corresponding part of the reference answer as the reason for the model's incorrect answer.

[0134] Step S502: Output the referee result in a preset format. For better subsequent viewing and analysis, the referee result of this embodiment is output with the suffix of xlsx, but the present invention is not limited thereto.

[0135] In this embodiment, the process of outputting the judgment result in step S502 is specifically as follows:

[0136] Step S5021: traverse all json data files in the input directory;

[0137] Step S5022: formatting the answers of the fiscal and taxation model to different categories of data in the data set according to the requirements of different categories of answers;

[0138] Step S5023: Judge the results of the tax model and the reference answers, and judge them one by one in different dimensions and steps according to the prompt word rules;

[0139] Step S5024: normalize the number of stars in different dimensions in the results after judging according to the weights of the different dimensions;

[0140] Step S5025: Map and output the normalized results in the form of "partially correct, correct, wrong", and perform statistics on the data set results;

[0141] Step S5026: Convert the data processed by the above steps and the detailed data field results into binary format and output them in xlsx format.

[0142] After obtaining the questions of the fiscal and taxation data set, the reference answers, the answers of the fiscal and taxation model, and the prompt words, the most critical step is the automated evaluation. The accuracy of the results of the automated evaluation lies in the binding strength of the prompt words. For the models in the fiscal and taxation field, this embodiment sets the four dimensions of accuracy, relevance, completeness, and coherence to gradually analyze the number of stars obtained in the answers of the fiscal and taxation model in different dimensions. For the results with incorrect answers, the referee model needs to find out which specific step the error occurred in, and analyze the wrong place, so as to provide effective help for the training and learning of the fiscal and taxation model. JSON output is performed by obtaining the number of stars in different dimensions. And normalization is performed according to the weight coefficients of different dimensions, and the judgment of the referee model on the answer results of the fiscal and taxation model data set and the statistics of different results are output. In order to better analyze the results of the automated evaluation in the future, it is necessary to traverse the file, and convert and map all important attributes of each data 'question, answer, reference answer, referee result, dimension star number, final score' to obtain the final json result, and finally assemble the data, convert the file format, and finally convert it into a table data file for output.

[0143] Compared with CN117785664A mentioned in the background technology, the present invention reduces the data evaluation in terms of dimensions, and focuses on the dimensions of the professional aspects of the data set. It uses the professionalism of the financial and taxation data set to extract the three professional dimensions of accuracy, completeness and coherence to evaluate the accuracy and smoothness of the model answer results. Due to the extraction and adoption of this professional data set, the financial and taxation field data set evaluation method based on OpenCompass has a good model answer result in the data set evaluation in the financial and taxation field. Compared with CN117911179A mentioned in the background technology, since CN117911179A requires verification data for review, this verification data has a strong constraint and cannot meet the needs of data diversity. And because the present invention uses the model answer result to first compete with other model results, and then combines the unique regular matching principle to output the data results. Therefore, compared with CN117911179A, it not only has multiple dimensions of evaluation indicators to improve the accuracy of the model's answers, but also obtains professional results based on model adversarial and unique regular matching rules.

[0144] The method for evaluating a data set in the field of finance and taxation based on OpenCompass provided by the present invention solves the effect of automatically evaluating the training model and reduces the manual audit cost after the model answers of a large number of data sets. For professional data sets in the field of finance and taxation, the temperature coefficient and the maximum output and input parameters applicable to the data set in the professional field are configured. Since different nouns in finance and taxation imply the size of their tax rates and related tax exemptions, Chinese capital numbers and the smallest unit, etc. are actually mapped, the model results are more conducive to the objective scoring of the referee model. And due to the debit / credit relationship in the professional answer format requirements in the accounting entries, it is necessary to configure the prompt words of the judgment model to allow it to be relatively objectively scored in different dimensions such as accuracy, relevance, coherence and completeness, and finally obtain the results of the automatic evaluation according to the coefficient method of different dimensions. Then, by matching the model answer result rules for different categories of data in the data set, judging the model answer and the reference answer by whether there is an implicit conditional conversion of the debit and credit relationship and the relevant tax rate in the accounting entries, the referee result is normalized and mapped according to the number of stars obtained in different dimensions, and the correct, partially correct and wrong model answer results are obtained, and the result statistics are performed. Finally, the important attributes in the dataset, such as questions, answers, reference answers, referee results, dimension stars, final scores, etc., are converted into JSON data and output in Excel format, thus realizing automated evaluation based on OpenCompass.

[0145] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0146] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating a data set in the field of finance and taxation based on OpenCompass, characterized in that: include: Step S1: Configure the financial and tax model interface parameters for the OpenCompass evaluation tool; Step S2: Set up the financial and taxation data set and its overall automated evaluation process, including: Step S201: According to the finance and taxation field, finance and taxation related questions that have been manually classified according to rules in advance and their reference answers are used to form a finance and taxation data set; Step S202: inputting the fiscal and taxation data set into the fiscal and taxation model to obtain the answer result of the fiscal and taxation model; Step S203: format the answer results of the fiscal and taxation model according to the classification requirements and output the final data set results; Step S204: scoring the data set results outputted in step S203 according to preset evaluation rules; Step S205: outputting the data set results and their corresponding analysis; Step S3: Establish prompt words for the financial and taxation model according to the type of questions in the data set results; Step S4: Format the answer results in the tax model data set; Step S5: Based on the OpenCompass evaluation tool and according to the content of the model prompt words, the results of the financial and tax model data set are evaluated one by one.

2. The method for evaluating a data set in the financial and taxation field based on OpenCompass according to claim 1 is characterized in that: Step S1 specifically includes: Step S101: Create a fiscal and taxation model file in the models folder, configure the method for retrieving fiscal and taxation model interface parameters in the fiscal and taxation model file, and then configure the model request parameters; wherein the model request parameters include the fiscal and taxation model custom class name, model name, access permission token, and model address; Step S102: adjusting the temperature coefficient required by the tax model data set, generating the highest cumulative probability parameter of text quality and the model constraint parameter; Step S103: Set the maximum number of generated tokens and the maximum input length.

3. The method for evaluating a data set in the financial and taxation field based on OpenCompass according to claim 1 is characterized in that: During the transmission of the fiscal and taxation data set in step S202, corresponding fiscal and taxation classification fields are set for different data classifications, and the value of each fiscal and taxation classification field is matched with the model prompt word through a dictionary to find the model prompt word for the corresponding classification question; then the fiscal and taxation model finds the corresponding model prompt word according to the classification of the data to carry out problem thinking and generate answers to normative questions; finally, the answer result of the fiscal and taxation model is provided for the input fiscal and taxation data set.

4. The method for evaluating a data set in the financial and taxation field based on OpenCompass according to claim 1 is characterized in that: Step S203 is specifically as follows: the answer results of the financial and taxation model are first screened for data validity according to the classification requirements, and then segment breaks are inserted according to the characteristics of the results of the financial and taxation calculations and accounting entry questions to form formatted JSON data and then output the final data set results, where the data set results include: classified questions, reference answers to the questions, and the answers to the questions by the financial and taxation model.

5. The method for evaluating a data set in the financial and taxation field based on OpenCompass according to claim 1 is characterized in that: The specific formatting process of step S4 for the accounting entry field is as follows: Step S401: Screen the answer results in the accounting entry field in the tax model data set and divide them into concept questions, accounting entry questions without calculations, and accounting entry questions with calculations; Step S402: For concept questions, obtain valid content through preset identifiers, and then format non-text characters and line breaks through the invalidFilter function to output the content that needs to be retained; For accounting entry questions without calculations, first execute the processing of concept questions, then use the pattern_accounting function to format the debit and credit relationships in the answer and output the processing results; For accounting entry questions that require calculations, first map and match the implicit tax rate keywords with the tax rate taxMap object, then format the calculation process into separate calculation steps, and then execute the processing of accounting entry questions that do not require calculations.

6. The method for evaluating a data set in the financial and taxation field based on OpenCompass according to claim 1 is characterized in that: The specific formatting process of step S4 for the tax field is as follows: Step S411: Screen the answer results in the taxation field in the fiscal and taxation model data set and divide them into basic policy type questions and calculation type questions; Step S412: For basic policy type questions, obtain valid content through preset identifiers, and then format non-text characters and line breaks through the invalidFilter function to output the content that needs to be retained; For calculation type questions, first map and match the implicit tax rate keywords with the tax rate taxMap object, then format the calculation process into calculation steps, and then execute the processing of basic policy type questions. Among them, when the data set contains Chinese numerals, it is also necessary to convert the text and numbers.

7. The method for evaluating a data set in the financial and taxation field based on OpenCompass according to claim 1 is characterized in that: Step S5 specifically includes: Step S501: Compare and judge the reference answer to the question and the answer of the fiscal and taxation model from different dimensions. If the consistency between the two reaches the preset threshold standard, it is determined to be a correct answer, and the corresponding question is rated with a star rating, and the reason for the rating is given; for the answers that are determined to be inconsistent, the inconsistent content parts are marked, and the error cause analysis and star rating are provided; Step S502: Output the referee result according to a preset format.

8. The method for evaluating a data set in the financial and taxation field based on OpenCompass according to claim 7 is characterized in that: The part of the content that is identified as inconsistent in step S501 and the analysis of the cause of the error are specifically as follows: the inconsistent content is identified and extracted from the result of the financial and tax model answer, and the explanation of the corresponding part of the reference answer is used as the reason for the wrong model answer.

9. The method for evaluating a data set in the financial and taxation field based on OpenCompass according to claim 7 is characterized in that: The specific process of outputting the judgment result in step S502 is as follows: Step S5021: traverse all json data files in the input directory; Step S5022: formatting the answers of the fiscal and taxation model to different categories of data in the data set according to the requirements of different categories of answers; Step S5023: Judge the results of the tax model and the reference answers, and judge them one by one in different dimensions and steps according to the prompt word rules; Step S5024: normalize the number of stars in different dimensions in the results after judging according to the weights of the different dimensions; Step S5025: Map and output the normalized results in the form of "partially correct, correct, wrong", and perform statistics on the data set results; Step S5026: Convert the data processed by the above steps and the detailed data field results into binary format and output them in xlsx format.

Images